Issuu on Google+

G OV E R N A N C E and ANTICORRUPTION A S S E S S I N G R E S U LT S I N P R O J E C T S

http://gacinprojects

This study is one in a series of papers that describe project-level approaches to governance and corruption October 2010issues, identify their results, and draw lessons that other task teams may be able to use.

Case Study 4 How can the quality of education be improved in an environment where

RECURSO/REACT: An Effective Strategy for Improving Educational Achievement

there are no standards and accountability mechanisms are weak? How can citizens be moved from inertia to action? The Government of Peru and the World Bank addressed these questions by developing analytic studies (RECURSO) followed by development policy lending (REACT). The program’s strategy aimed to bring together citizens and service providers, not only by engaging them but also by raising their expectations and incentives. As a result of this program, whose monitoring is now part of the government’s performance budget, the percentage of second-grade children with age-

The Program RECURSO (REndición de CUentas para la Reforma SOcial)—a series of analytic studies financed by the World Bank and the United Kingdom’s Department for International Development (DFID)—aims to strengthen accountability and results in education, health, and nutrition services by diagnosing the main problems in those sectors and making relevant recommendations. Responding to the RECURSO recommendations, the World Bank developed a series of development policy loans, REACT (Results and Accountability) DPLs, whose education objectives are to improve outcomes in second-grade literacy by providing up-to-date information on the performance of schools and individualized data for parents on the learning status of their children. The operation also supports actions to reduce exclusion, rationalize programs, improve targeting, and increase the participation of poor communities in budget processes and program monitoring. The total cost to date of the IBRD-supported RECURSO analytic and advisory activities is about US$2.6 million. The REACT series has provided a total of US$480 million in IBRD budgetary support so far through two operations, and a third operation is under implementation.

appropriate comprehension

Diagnosis of the Problem

G OV E R N A N C E and ANTICORRUPTION

rose dramatically between 2004

and 2009.

In the early 2000s the World Bank helped the Government of Peru reach high levels of education coverage and put in place transparency and accountability measures, yet several studies showed that the quality of education services remained low. In 2005 only 15 percent of Peru’s second-grade students were reaching an appropriate level

A S S E S S I N G R E S U LT S I N P R O J E C T S

http://gacinprojects

This study is one in a series of papers that describe project-level approaches to governance and corruption issues, identify their results, and draw lessons that other task teams may be able to use.


GOVERNANCE and ANTICORRUPTION ASSESSING RESULTS IN PROJECTS

Table 1  Governance Measures and Results at a Glance Problems identified

Governance measures

Results

• Poor quality of services in education • Weak accountability framework • Lack of Government support • Low expectations • Weak results culture and scrutiny by civil society

• Transparency and communication strategy • Community and CSO participation and monitoring of results • School management committees • Budget-friendly consultation • Participatory budget

• 30% of second grade children in level 0 in 2007 and 23% in 2009 as part of UST • 16% of second grade children in level 2 in 2007 and 23% in 2009 • 80% of Peruvians approve the program

of literacy, and roughly half were unable to read at all.1 As part of the RECURSO work, the Bank found that of 136 randomly selected secondgrade children, 35 percent were not able to read a single word. This situation had broad implications: poor early reading performance contributes to low and unequal performance later on, and in the long run it could affect Peru’s ability to compete in the global economy. Several analyses explained that while users and community leaders contributed to expanding the coverage of services—they built schools, hired teachers, and lobbied for improvements to school infrastructure—they did not push for improving the quality of these services; indeed, 80 percent of parents surveyed by the Bank were pleased with the quality of their child’s school. The Bank found that while it is easy to see and measure coverage (every parent knows if there is a school in town), it is difficult to see or measure quality. The lack of clear quality benchmarks prevented parents from demanding better education for their children.

2

Peruvian society lacked not only standards, but also a system of accountability pressure and systems built around learning. For example, most teachers did not expend the hours of effort required to do their jobs well unless they were supervised and motivated by both communities and bureaucracy, and unless expectations were made clear. Thus, as a study in the Junin region found, low-quality outcomes were found not only in schools serving the poorest communities, but also

in well-endowed schools (urban schools) receiving substantially more resources; it concluded that the root of these deficiencies is poor management, not just resource constraints.2 RECURSO findings were incorporated in one main publication3 and, more importantly, were communicated through videos, targeted brochures, and other means. Almost all the RECURSO recommendations were taken up by the political parties, and the Government of Peru requested the Bank to provide a series of loans (REACT) to develop its strategy.

Governance Measures The Government took a two-pronged approach to the education issues, addressing both supply   In 2004 the Government’s Quality Measurement Unit (UMC) classified second-grade students into four groups based on attainment levels (0, 1, 2, and 3). A student in group 0 cannot read and comprehend even the simplest sentence; at Level 1 the child can read and comprehend a simple sentence; at Level 2 the child can read a more complex paragraph but without full comprehension. By the end of second grade, a child should reach Level 3: the ability to read and fully comprehend a more complex paragraph.. The UMC classified 46 percent of second-grade students nationwide in Level 0, 16 percent in Level 1, 23 percent in Level 2, and 15 percent in Level 3. 2    See Cotlear, D., and Kudo, I. “What can a Regional Government do to Improve Education? The Case of Junin Region, Peru.” World Bank, 2008. 3    See Cotlear D. ed., A New Social Contract for Peru, World Bank, 2006. 1 


Case Study 4

RECURSO/REACT

(education governance and service providers) and demand (parents and other citizens). The strategy it developed with the Bank was characterized by the good governance principles of transparency, accountability, and participation and was based on three pillars: standards, accountability systems, and Government support.

Standards The Government established the standard that all grade-2 children should attain age-appropriate reading comprehension (that is, reach Level 2 on the national test). In 2006 the Government, supported by REACT loans, established a universal standard system to test reading attainment at the end of second grade; provide each school, child, family, and community with information about performance; and use that information to set schoolspecific performance goals. The assessment for second grade was first carried out in December 2006 and was repeated in 2007 and 2008.

Accountability The universal system testing (UST) results are not being used only to track outcomes: the fundamental idea is to establish a school-level accountability system that will support a transformation of outcomes. The results of the 2007 tests, including specific reports for each region and “report cards” for each school and child, were disseminated through the local education offices in mid-2008. In 2009, as a further step, UST results were delivered directly to schools, and a media campaign was organized to advise parents to request their child’s scores and encourage schools to discuss their results and target improvements. The aim is to strengthen the feedback process and ensure that local school systems use the results of the testing system to inform their school improvement strategies. As part of REACT, the Ministry of Education commissioned a consultant to pilot approaches to discussing the outcomes among schools, parents, and communities and supporting decisions about ways

Box 1  The Case of Solaris Solaris, an international NGO in Peru, conducted a very successful experiment in six public schools it managed. Solaris first tested pupils’ ability to read and found that reading fluency was very low. They then used a RECURSO video to motivate the development of a plan to measure reading fluency, and developed specific targets and proposed instruments. After only three months a retest found remarkable progress in reading fluency (all schools improved by an average of 30%) and reading comprehension (all schools improved by an average of 43%). Solaris’s strategy: 1. Learning goals were incorporated in schools’ strategies. 2. The schools established a reading time for all primary school grades that included guided reading practice; reading in pairs between higher-grade children and students from lower grades; reading for pleasure; and word games. 3. By involving parents in reading practice, the schools informed them about the use of standards and the application of the reading measurement. 4. The schools upgraded their libraries and promoted lending books for children to take home. 5. Twice each semester the schools continue to evaluate children’s grade and reading abilities against the agreed standards. Source: Elaboration from Solaris video.

to improve outcomes next time around. The results of this pilot study will feed into further strengthening of the national dissemination strategy for the 2009 USTs, for implementation in the 2010 school year. The strategy also seeks to empower parents through, for example, parents’ associations, where parents can meet and discuss children and school performance. Another important feature of the accountability system is Peru’s national quality certification

3


GOVERNANCE and ANTICORRUPTION ASSESSING RESULTS IN PROJECTS

system, SINEACE (Sistema Nacional para la Evaluación, Acreditación y Certificación de la Calidad Educativa), established in 2006. This independent entity evaluates and accredits educational institutions at all levels: basic education, universities, and non-university higher education. One of SINEACE’s main aims is to improve teacher training, which will make an important contribution to education quality. The Bank and the Government are also working to strengthen the accountability system through decentralization, institutionalizing the delivery of more resources to the school level and greatly strengthening support to help teachers improve their classroom effectiveness. The local education offices are playing a major role in teacher supervision and training, and the Ministry of Economy and Finance is empowering schools to address their own needs by providing maintenance budgets for their direct execution. Figure 1 illustrates the accountability system Peru’s Government is putting in place in the education sector.

Support Accountability pressure built around standards can lead to improved results only if all actors have all the information and skills they need to meet the standards. Thus, support is the third important pillar in quality improvement. To help strengthen support systems, in January 2007 the Ministry of Education completed a national mandatory evaluation of teachers’ capacities (with participation by 65 percent of all teachers, in spite of union opposition). The Ministry has implemented training to address the weaknesses identified in the evaluation, including training on teaching reading and math in the early grades, and has designed workbooks for reading and math in grades 1 and 2. The Government also changed the incentives related to Peru’s education sector. For example, the congress approved a new law that linked teachers’ pay to evaluation. Teachers opposed the new law and went on strike, but for the first time ever public opinion sided with the Government and the legislation passed. As another example, the

Figure 1  Accountability system Politicians and policymakers (which may be at subnational level as well as national), including executive and legislative Expectations of performance Information on delivery against expectation

Direct voice, taxes, votes Feedback on progress

Governance, watchfulness, choice over providers, including opinion on teacher performance and progression Citizens and more specifically, parents at schools

4

Source: A New Social Contract for Peru.

Service provided (children educated), information on service provision

Policy implementers (Ministry, DREs, UGELs)

Information on performance to standards

Norms and standards, specific goals, resources, promotion and pay progression support in coming to standard

Actual service providers (schools)


Case Study 4

RECURSO/REACT

Government established a prize for the 100 best rural teachers whose second-grade students read at the level of the standard (an idea that the World Bank later exported to other countries). The Government of Peru is tracking the outcomes from RECURSO and REACT as part of the performance budget system in the Ministry of Economy and Finance. In fact, the RECURSO/REACT programs have supported the Government’s performance budgeting strategy, which incorporates strategic goals for improving reading attainment and sets expected outcomes and goals for 2011. For each strategic program, the budgetary system has been analyzed to determine the relevant financing sources and service providers. For education these sources are the Ministry of Education, the Presidency of the Council of Ministers, and the regional governments. The performance budget process identifies the corresponding budget line items, and physical output targets have been established for each budget line at the national, regional, and local levels. A baseline has also been established for each program’s high-level outcomes, and progress on expenditures and intermediate physical outputs is being monitored quarterly for each executing agency—in education, the USTs are being used to monitor students’ attainment. The World Bank is working with regional governments and the Finance Ministry to promote additional funding for strategic activities in education, such as strengthening the teacher support system, to help teachers improve their classroom effectiveness.

Other Important Components Three aspects of the Government’s strategy did much to facilitate the engagement of stakeholders and thus the success of the program: a comprehensive and innovative communications program, a transparency measure, and participatory local budget processes.

Communications zz

zz

zz zz

zz

RECURSO developed an education video that described the RECURSO findings and explained the importance of the evaluation for parents and teachers, showing them how to use the results. The video challenged parents to find out how well their children are reading, told them that they have the right to demand a good education, and provided simple, clear, and measurable expectations about how children should learn, grow, and be healthy. Moreover, it created a strong impression on the government: the Government also asked the Bank to produce TV and radio spots to be shown during regular TV hours. Three books summarizing the main RECURSO findings and recommendations were published. In addition, brochures were targeted at midlevel sector officials. A dedicated website provides access to all materials produced by the team. A media campaign was designed and implemented to engage TV, radio, and newspapers in providing publicity for the main messages. One of its products was a radio mini-series about education standards and parental empowerment vis-à-vis the schools, produced in Spanish and translated into Quechua, Aymara, and Asháninka. The Government engaged with stakeholders— Government officials, civil society organizations, subnational authorities, and donor organizations—throughout the preparation of the REACT loans. In addition, an education forum—comprising school managers, academics, politicians, and education experts— was developed to create a space for reflection.

These innovative ways of communicating and engaging communities and political parties led to a national consensus, shared by the government, civil society entities, and the private sector, and reflected in the media, on the need for action to transform early-grade education outcomes.

5


GOVERNANCE and ANTICORRUPTION ASSESSING RESULTS IN PROJECTS

Transparency. Another important component of the Government’s strategy is the effort to make social spending more transparent, equitable, and effective. The Ministry of Finance has established the transparency portal Consulta Amigable (“Friendly Consultation”) on the Integrated Financial Administration System. This portal provides information on economic indicators and projections; summaries of the strategic plans of different sectors; central, regional, and municipal government budgets and expenditures; financial statements of all public agencies; and information on internal and external public debt. The Government publishes the modified budget in Consulta Amigable, facilitating civil society monitoring of budget implementation. Participatory local budgeting. As part of

decentralization to regions and municipalities, the government has promoted participatory budgeting and monitoring in local and regional governments. The aim is to make services more accountable to citizens and users through the publication of timely and reliable budget information. Civil society entities are actively monitoring budget expenditures.

Results The results of RECURSO/REACT are already impressive. Follow-ups to the 2006 assessment of second-grade children found that the proportion of students classified in Level 0 was down from 46 percent to 30 percent in 2007, and to 23 percent in 2009; at the same time, the percentage of children in Level 2 rose from 16 percent in 2007 to 23 percent in 2009. Given the broad national scope of Peru’s interventions on education, the outcomes supported by RECURSO and REACT are not exclusively attributable to the World Bank’s analytic and advisory activities, technical assistance, and development policy lending. However, the program is certainly leading to a stronger relationship between agents (parents) and service providers (see Figure 2). Moreover, the implementation of the program has built a wide consensus among governments and civil society, enabling reform in the education sector.

Figure 2  RECURSO/REACT Result Chain Government of Peru (and World Bank)

Transparency (communication strategy, Consulta Amigable)

Agents (parents)

Accountability and M&E (standards, school improvement plan, student tests, teacher performance, choice over providers, school management committees)

Service providers

Support (training, resources)

Performance-based budgeting with strategic goals on education 30% of second grade children in level 0 in 2007 and 23% in 2009, 16% of second grade children in level 2 in 2007 and 23% in 2009

46 % of second-grade children in Level 0 in 2004 survey.

6

Other Government and Bank-funded projects


Case Study 4

RECURSO/REACT

Lessons Learned Several major lessons can be drawn from the RECURSO/REACT experience and applied in similar environments

zz

Project Design and Diagnostics zz

Diagnosis of problems. RECURSO provided

Peru’s Government with a diagnosis of the main problems in education and recommendations on addressing these problems. Using international comparisons, RECURSO gave context for its discussion of the coverage, quality, and equity of education in Peru and its analysis of the evolution and distribution of public expenditures and of the service delivery institutions. RECURSO led to a national consensus—shared by the government, civil society entities, and the private sector, and reflected in the media—on the need for decisive action. zz

zz

Appropriate strategy and sequencing of actions. The program used a sequence of

actions that turned out to be very effective: (a) issuing simple standards of quality and setting quantitative targets to provide users and policymakers with instruments to measure progress in quality, (b) implementing clear lines of accountability that go beyond the hierarchy of the corporation and make providers accountable to users and to policymakers, and (c) investing in capacity building and in new incentives for providers willing to work in a framework of accountability to quality standards and targets.

zz

Project Implementation zz zz

Peru set up specific mechanisms—school management committees—to induce a more responsive relationship between communities, parents, and service providers, allowing both greater access to information and incentives and pressures from stakeholders to improve the system. This approach was crucial to raise expectations and Accountability

system.

zz

break the vicious cycle of pessimism and the blame-the-child or blame-the-society game that seemed to be pervasive in Peru. Government willingness. Governance reforms and national strategies to address poverty are effective only when they are supported by political will and the political capacity of governments to follow through on them; and creating political will and negotiating with governments takes much time and effort. Importance of “politics.” Bank staff deserved credit for being able to engage “politically”; recognizing that it is extremely hard to sustain difficult reforms without sufficient political support, Bank staff explored ways they could intervene to strengthen that support. Linkages between actors at different levels are of crucial importance: for instance, an international financial institution only includes a given principle in its country assistance plan if both the institution and the government want it— and government wants it only if the political parties do. Empowering stakeholders. The program empowered stakeholders in three ways: by giving them appropriate information, by telling them what this information is about and how to use it, and by creating avenues for them to improve the system. There are already some encouraging results in terms of involving communities around the dissemination of the results for the UST and engaging school communities (parents, teachers, and other relevant actors) in discussing how the results can be improved through their school improvement plans. Access to information is not enough. This case study shows the importance not only of accessing simple and appropriate information (i.e., standards) but also of creating avenues for beneficiaries to complain and improve the system (i.e., meetings with teachers). Communication tools. Audiovisual and other communication tools reached a wide audience, capturing the attention of politicians

7


GOVERNANCE and ANTICORRUPTION ASSESSING RESULTS IN PROJECTS

and communities and helping to overcome the opposition of service providers.

Monitoring and Evaluation zz

Evaluation culture. Developing a culture of

who accept it in Peru (i.e., a salary increase for teachers who assess students) was extremely important. Individual school (and teacher) performance is being measured in terms of children’s learning results, and information about schools is being disseminated.

evaluation and creating incentives for those

References Arnold, Anne-Katrin. “From Inertia to Participation: The Case of RECURSO in Peru.” World Bank Blog, 2009. http://blogs. worldbank.org/publicsphere/inertia-participation-caserecurso-peru Cotlear, D. “Peru: Making Accountability Work – Lessons from RECURSO.” En breve, World Bank, 2008. Cotlear, D., and Kudo, I. “What can a Regional Government do to Improve Education? The Case of Junin Region, Peru.” World Bank, 2008. Cotlear, D., ed. A New Social Contract for Peru. An Agenda for Improving Education, Health Care, and the Social Safety Net. World Bank, 2006. Crouch, Luis. Toward High-Quality Education in Peru: Standards, Accountability, and Capacity Building. World Bank, 2007.

Tesliuc, Cornelia Mihaela, and Ian Walker. Social Safety Nets in Peru. World Bank, September 2007. World Bank, Results and Accountability (REACT) Development Policy Loan, Program Document, April 24, 2007. http:// imagebank.worldbank.org/servlet/WDSContentServer/ IW3P/IB/2007/05/15/000020953_20070515135957/ Rendered/PDF/39673.pdf World Bank, Second Results and Accountability (REACT) Development Policy Loan, Program Document, March 5, 2009. http://imagebank.worldbank.org/servlet/ WDSContentServer/IW3P/IB/2009/03/23/000334955_20 090323033833/Rendered/PDF/471040PGD0PE0P101O fficial0Use0Only1.pdf World Bank, “Education in Peru.” http://www.youtube.com/ watch?v=-BxL1aqb6mY

G OV E R N A N C E and ANTICORRUPTION

This study was prepared by Tommaso Balbo di Vinadio under the supervision of Ivor Beazley and

8

NOITP

with the assistance of several World Bank staff: Ana Maria Muñoz Boudet, Daniel Cotlear, Ian Walker, Inés Kudo, and William Reuben.

A S S E S S I N G R E S U LT S I N P R O J E C T S

http://gacinprojects stcejorpnicag//:ptth

STCEJORP

This study is one in a series of papers that describe project-level approaches to governance and ,secorruption ussi noitpuissues, rroc dna ecnanrevog ot sehcaorpp identify their results, and draw lessons that other task teams may be able to use. .esu ot elba


Recurso peru