Addressing determinants of provider performance

learning materials. Likewise, school principals need to coordinate activities and connect with teachers and students virtually. Existing survey tools will need to be adapted further to capture these and other novel aspects of ICT that can shape learning outcomes. With this in mind, the World Bank Group’s Education Global Practice has been developing the EdTech Readiness Index, composed of eight indicators, including connectivity, teacher training, digital learning resources, and online assessments. This initiative and the SDI team’s ongoing collaboration with the Education Global Practice could create additional synergies in the future. Similarly, to the extent that COVID-19 itself continues to pose a challenge, new physical requirements for spaced seating; water, sanitation, and hygiene infrastructure; and protective equipment will become essential elements of school life.

The performance of individual service providers (such as frontline health care workers and teachers) is a crucial determinant of the quality of services that citizens receive. What factors shape how well frontline service professionals perform, and what levers exist to support providers in doing their best work? SDI instruments are evolving to deliver new evidence on why providers make certain good or suboptimal choices in their daily practice and what the role of management is.

Better understanding providers’ performance

In health, the current SDI vignettes capture whether a provider gives the correct diagnosis and treatment, and are being expanded to provide more information on why a provider is diagnosing a condition incorrectly. For example, although the existing data show that many providers are not effective at diagnosing diabetes, it is not clear whether they think that the patient is healthy or instead is displaying risk factors for another disease, such as depression or chronic fatigue, that might present similarly to diabetes. At present, health care providers can generally diagnose the tuberculosis vignette quite well, but they do poorly with treatment. Based on consultation with medical experts, the questionnaire now includes more of the most common incorrect answers as well as the requirement to record any nonstandard answers given. This information will shed light on the most common errors and will help to provide more pinpointed recommendations to improve provider performance.

In schools, teacher observation captures more granular information about time spent on different activities by a teacher and students during a class

period. The SDI methodology that is used to gather this information closely follows the widely applied Stalling’s Classroom Observation System, in which a 30- to 60-minute period is observed and, for each minute, the enumerator notes a “snapshot” of current activities in the classroom. The instrument allows an enumerator to record what every person in the classroom is doing, including whether they are engaged in a learning activity or not. The information gathered is used to construct the time-on-task indicator, which refers to the amount of time during a class period in which a teacher is actively engaged in teaching or students are actively engaged in assigned learning tasks. Analysts can combine this indicator with data from the staff roster module (used to measure absence rate), the classroom observation module, and reported teaching hours to estimate the time spent teaching per day.

Other classroom observation tools that take advantage of new technologies have been used in more recent SDI surveys. For instance, the TEACH tool, developed by the Education Global Practice, was tested for the Mozambique SDI survey in 2018, in place of the standard classroom observation module. TEACH is similar to the SDI tool but generally involves recording classes on video and then having a trained observer view the videos and provide an assessment.5 In addition to time on task, this approach assesses the quality of teacher practices through behavioral information observed and classified into three categories: classroom culture, instruction, and socioemotional skills. Recent efforts on this front are trying to leverage machine-learning techniques to identify teacher-student interactions and engagement automatically through data mining of videos of classroom observations (see, for instance, Aung, Ramakrishnan, and Whitehill 2018). If proven reliable, these methods might help to reduce the cost of data collection and increase the information available to inform policies that promote effective teaching and learning in classrooms.

The role of management

Limited competence and high absenteeism among service providers can reflect many underlying factors, including the incentives that agents face within an organization, the design of which can have both positive and negative effects on the motivation and effort of employees (Ashraf, Bandiera, and Jack 2014; Björkman Nyqvist et al. 2019; Karachiwalla and Park 2017; Rasul and Rogger 2018). Intrinsic motivation also matters for a provider’s performance (Ashraf, Bandiera, and Lee 2018; Deserranno 2019), as do the financial resources that organizations and staff must work with and the selection of staff members in the first place (Das et al. 2013; Deserranno 2019). Together with these factors, which interact with each other (Donato et al 2017), growing empirical evidence shows that management practices have an important role to play in how organizations perform (Bloom, Sadun, and Van Reenen 2016).

The explanatory power of management carries over into the space of public service delivery. For example, Rasul and Rogger (2018) find that bureaucratic autonomy in Nigeria increases project delivery rates (although increasing incentives and monitoring among bureaucrats has the opposite effect). The quality of management (such as organizational practices in schools) has also been shown to predict educational outcomes (including math or other assessment results) within countries such as Brazil, Canada, India, Italy, Sweden, Uganda, the United Kingdom, and the United States (see Bloom et al. 2015; Crawfurd 2017; Di Liberto, Schivardi, and Sulis 2015). Interventions aimed at improving the quality of management in schools have also been shown to improve student performance, although the effectiveness of such measures varies by context (see Blimpo, Evans, and Lahire 2015; Fryer 2017).

The recent education SDI survey in Indonesia was coupled with the implementation of the Development World Management Survey (D-WMS) among school directors to assess the quality of school management. The D-WMS tool was created in 2008 as an adaptation of the original World Management Survey and covers 15 questions across the following five main areas: leadership, operations management, performance monitoring, target setting, and people management. Through an interview-style questionnaire aimed at school administrators (either principals or their second-in-command), trained and certified enumerators assign scores ranging from 1 to 5 along the five main areas. Again, preliminary results are included in the 2020 SDI report for Indonesia, but, because of the richness of the data, their link to the quality of service provision and student learning will be analyzed further in a separate publication. This methodology will continue to be replicated in future SDI surveys whenever possible.

Existing evidence suggests that management practices also affect quality of care in the health sector (Macarayan et al. 2019). Effective health care management requires oversight of many facets of service delivery, including the facility’s layout and patient flow, development and standardization of protocols, staff recruitment and retention, effective use of personnel and equipment, setting of targets, continuous tracking of performance, and use of accountability systems with performance incentives. Although many of the inputs into the provision of quality care, such as provider training or medical supplies, may be costly, management practices can often be improved in situations with constrained resources. When properly implemented, management interventions have the potential to be cost-effective.

A forthcoming survey in Kenya will contribute to the existing, though somewhat limited, evidence on the role of management practices in improving the quality of service delivery in health facilities. Specifically, a telephone survey will be implemented to assess the quality of management in a sample of hospitals and lower-level primary health care facilities that were included in Kenya’s 2018 SDI survey. The sample includes both public and private