Mapping Innovative Learning and Performance Management Approaches for Frontline Health Workers

Mapping and Assessing Learning and Perfomance Management Approahes for Frontline Health Workers 26 March 2021

Mott MacDonald 10 Fleet Place London EC4M 7RB United Kingdom T +44 (0)20 7651 0300 mottmac.com

Mapping and Assessing Learning and Perfomance Management Approahes for Frontline Health Workers 26 March 2021

Mott MacDonald Limited. Registered in England and Wales no. 1243967.

Registered office: Mott MacDonald House, 8-10 Sydenham Road, Croydon CR0 2EE, United Kingdom

Mott MacDonald | Mapping and Assessing

i

Classified as Internal

Issue and Revision Record Revision

Date

Originator

Checker

Approver

1.0

22 Jan 2021 Lucy Palmer

Ken Grant

James Fairfax

2.0

18 Feb 2021 Lucy Palmer

Sarah Passow

James Fairfax

Description

Document reference: Information class: Standard This document is issued for the party which commissioned it and for specific purposes connected with the abovecaptioned project only. It should not be relied upon by any other party or used for any other purpose.

We accept no responsibility for the consequences of this document being relied upon by any other party, or being used for any other purpose, or containing any error or omission which is due to an error or omission in data supplied to us by other parties.

This document contains confidential information and proprietary intellectual property. It should not be shown to other parties without consent from us and from the party which commissioned it.

Mott MacDonald | Mapping and Assessing

ii

Classified as Internal

Contents Executive summary

5

1

Introduction

10

2

Evidence on Healthcare Provider Performance

12

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12

12 13 15 17 18 19 23 23 24 25 26 27

3

4

5

Healthcare Provider Performance Evidence from Systematic Reviews Training attributes associated with effectiveness Evidence on CHW Training Evidence on Digital Approaches to Supporting Healthcare Providers Specific digital learning approaches Performance Theory Collaborative Quality Improvement Supportive Supervision Payment for performance Data for Accountability and Recognition CHW Performance and Community Involvement

Enabling Principles for Innovative Performance Strategies

29

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9

Using blended and layered approaches Using different media and formats tailored to context Designing with user needs in mind Incorporating problem-solving approaches Communities of practice and peer support Co-production and government buy-in Recognition, accreditation and lifelong learning Feasibility / Acceptability of Technology Measurement, Individual learning and Incentivisation

29 31 32 33 34 35 36 37 41

Future Trends and Emerging Technology and Innovation

45

4.1 4.2 4.3

45 46 48 48 50

Virtual reality and augmented reality Serious Gaming and Gamification Remote learning and COVID adaptations in other sectors 4.3.1 Education Sector and COVID Adaptations 4.3.2 Agriculture sector

Country Case Studies

53

5.1 5.2

54 57

Pakistan Ethiopia

Mott MacDonald | Mapping and Assessing

iii

Classified as Internal

5.3 5.4 5.5 5.6 5.7 5.8

6

7

8

Somalia Rwanda DRC Senegal Indonesia Summary

59 62 64 66 67 69

Donors/ Funders

73

6.1 6.2

73 75

Bilaterals Foundations/ Innovation Funders

Sustainability

77

7.1 7.2 7.3

77 77 78

Sustainability Landscape Global public goods and potential market shaping / funding models Actions Gavi can support

Strategic Opportunities for Gavi

81

Annex 1 - Key Informants List

84

Annex 2 - Bibliography

86

Annex 3 - Scoring Criteria

91

Annex 4 – Separate Document

92

Annex 5 - Global Platforms

93

Mott MacDonald | Mapping and Assessing

Classified as Internal

Abbreviations AFD

French development Agency (Agence Française de Développement)

ASHA

American Speech-Language-Hearing Association

BMGF

Bill and Melinda Gates Foundation

BMJ

The British Journal of Medicine

CHA

Community Health Academy

CHTK

Community Health Toolkit

CHW

Community Health Workers

cMYP

Comprehensive Multi Year Plan

CPD

Continuous Professional Development

CQI

Collaborative Quality Improvement

DFAT

The Department of Foreign Affairs and Trade

DHA

Digital Health Activity

DHGG

Digital Health Global Goods

DHILC

Digital Health Innovation and Learning Centre

DIAL

Digital Impact Alliance

ECHO

Extension for Community Healthcare Outcomes

EHR

Electronic Health Records

EIR

Electronic Immunisation Registry

EPI

Expanded Programme on Immunisation

FCDO

Foreign, Commonwealth and Development Office

FHIR

Fast Health Interoperability Resources

FHW

Female Health Workers

FITL

Framework for Immunisation Training and Learning

FLW

Frontline Health Worker

GNI

Gross National Income

GPS

Global Positioning System

HCPPR

Health Care Provider Performance Review

HIC

High Income Countries

HIE

Health Information Exchange

HNQIS

Health Network Quality Improvement System

HRH

Human Resources for Health

2

Mott MacDonald | Mapping and Assessing

Classified as Internal

HSS

Health Systems Strengthening

IA

Immunisation Academy

IAI

Interactive Audio Instruction

ICCM

Integrated Community Case Management

ICT

Information and Communication Technology

IMCI

Integrated Management of Childhood Illness

IMS

Incident Management System

IP

Internet Protocol

IPC

Infection, Prevention, and, Control

IRD

Interactive Research and Development

IRI

Interactive Radio Instruction

ITCH

Immunisation Training Challenge Hackathon

IVR

Interactive Voice Response

KWTRP

The Kenya Medical Research Institute

LEAP

The Lending for Education in Africa Partnership

LIFE

Life-Saving Instruction for Emergencies

LMH

Last Mile Health

LMIC

Lower- and middle-income countries

LMS

Learning Management System

mHealth

mobile health

MNO

Mobile Network Operators

MNO

Mobile network operators

MOH

Ministry of Health

MOOC

Massive Open Online Courses

MSAS

Ministry of Health and Social Action (Ministère de la Santé et de l’Action Sociale)

NADRA

National Database & Registration Authority

NHS

National Health Service

OSCE

Observed Structured Clinical Examination

OSCE

Objective Structured Clinical Checklist Examination

OSS

Open Source Software

P4P

Payment by Performance

PAR

Participatory Action Research

3

Mott MacDonald | Mapping and Assessing

Classified as Internal

PIH

Partners in Health

PITB

Punjab Information Technology Board

PNA

National Supply Pharmacy (Pharmacie Nationale d’Approvisionnement)

PSSD

Strategic Digital Health Plan

RCCE

Risk, Communication and Community Engagement

RCT

Randomized Controlled Trials

RISE

Rapid Immunization Skill Enhancement

SDG

Sustainable Development Goals

SEARO

South East Asia Regional Office

SGG

Serious gaming and gamification

SIA

Supplementary Immunization Activity

SRCS

Somali Red Crescent Society

TCO

Total Cost of Ownership

TCP

Transmission Control Protocol

TdH

Terre des Hommes

TGLF

The Geneva Learning Foundation

TIH

The Indus Hospital

UHC

Universal Health Coverage

VLE

Virtual Learning Environment

WHO

World Health Organization

4

Mott MacDonald | Mapping and Assessing

Classified as Internal

Executive summary Gavi has contracted Mott MacDonald to undertake a mapping and assessment of learning and performance management approaches of frontline health workers. The purpose is to inform Gavi’s support to countries during its new 5.0 strategy, to build health workforce capacity to address the problem of high unreached (zero dose) children and under-vaccinated populations, as well as preparing health systems for COVAX and other vaccine introduction.

Since the early 2000s, there have been large increases in donor financing of human resources for health (HRH) - currently estimated at estimated at $4-6 bn per annum1 - yet much investment is not coordinated with national health workforce plans and there has been little analysis of impact. COVID-19 has brought about not only new imperatives but also opportunities to make use of new technologies to disrupt the way things are done. The need to limit face-toface interaction, upskill large numbers quickly and cope with even more pronounced economic pressures worldwide speak to the need for more efficient and impactful approaches.

Focusing on performance and quality of care The Lancet Commission on High Quality Health Systems asserts that moving to a high-quality health system is primarily a political, not technical, decision. Governments should start, it suggests, by establishing a national quality guarantee for health services, specifying the level of competence and user experience that people can expect, and against which governments would be held to account. We know from evidence reviews that knowledge is necessary but not sufficient for provider performance which is influenced by a range of factors including sufficient budgets, supervisory systems, staff motivation and retention, supply chains and back-up health facility readiness. Provider performance, therefore, cannot be considered in isolation from the wider context of health system reform, the need for better governance and accountability, and a more people-centred approach.

Content-driven, one-size fits all approaches are effective only for specific situations and need to be complemented by more learner-centred, layered approaches in others. Applying the evidence at scale becomes an implementation challenge, which is why health sector coordination, and combinations of approaches, is so important. Even the best, most innovative solutions will not be effective without appropriate workforce strategies, health financing, governance and health data in place.

Review of the most promising examples of scalable innovation suggests factors in success relate to the extent solutions are integrated into existing systems, institutional, behavioural and political contexts and resource environments. This suggests that broader, multifaceted, health systems approaches to performance (ultimately impacting quality of care and health outcomes) are needed.

Evidence from systematic reviews A recent systematic review of provider performance strategies confirmed that multifaceted strategies targeting infrastructure, supervision, other management techniques, training and group problem solving tended to have large effects. Combining training and supervision had larger effects, for example, than use of either strategy alone. Group problem solving including “collaborative quality improvement” strategies were shown to be particularly effective. 2 As for the training attributes most associated with effectiveness, educational outreach visits at health

1

Personal communication with Jim Campbell, WHO, 13 January 2020

2

Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review.

Alexander K Rowe, Samantha Y Rowe, David H Peters, Kathleen A Holloway*, John Chalker, Dennis Ross-Degnan. Lancet Glob Health 2018; 6: e1163–75. Published Online October 8, 2018

5

Mott MacDonald | Mapping and Assessing

Classified as Internal

workers’ sites were found to be more effective than in-service training, which was, on average, more effective than peer-to-peer training and self-study. In-service training that incorporated clinical practice tends to be more effective than training without it, and training that at least partly occurred at health workers’ routine worksite (on-site training) tended to be more effective than training that was entirely off-site. These findings likely relate to on-site training being more likely to be immediately relevant to a provider’s role with a greater focus on hands-on practical skill building (see Section 2.3). These are aspects that any digitalisation of training should try to replicate where possible.

Integral to any approach must also be effective monitoring of health worker performance and quality of care. This is especially true because of the highly variable effectiveness of strategies from one context to another, and the need to detect specific clinical/inter-personal aspects of service delivery that need improvement. Monitoring data can also be shared publicly to help accountability efforts needed to ensure high quality systems.

Best practice from implementation experience Some important principles of best practice for interventions to improve health worker performance, many somewhat overlapping, are as follows: ➢

Using a blended learning approach, where components can be layered and tailored to requirements, generally works best; finding the right balance between face to face and virtual approaches might need trial and error to get right;

➢

Using different media and formats tailored to context increases success of training

➢

Co-designing with health care providers themselves allows training to focus on practical issues rather than theory, thus becoming more relevant and immediately transferable to health providers’ practice

➢

Including problem-solving approaches and collaborative quality improvement addresses specific identified challenges rather than relying on approaches which deliver standardised content

➢

Group dynamics, peer support, buddies, and communities of practice are an important part of skill-building and norm setting

➢

Ensuring co-production with government is more likely to lead to ownership and government buy-in and sustainability

➢ Linking training with accreditation as part of a lifelong learning approach increases uptake /completion rates and ensures better alignment with national health workforce strategies.

These principles are expanded, and illustrated with examples, in the main report.

Use of digital approaches The above-mentioned systematic review of performance found that the average effectiveness of improvement strategies that included mHealth/ICT (information and communication technology) were not particularly high, with a median improvement of just 1 %-point for ICT alone, and 8 %points for ICT in combination with other intervention components. Some digital education has been conceptually pedagogically weak, with more attention to the technology than learning impact. However it is a fast-evolving field and WHO has recently published a discussion paper “Digital education for building health workforce capacity” with evidence pointing to digital education being at least as effective as traditional education in improving health professionals’ knowledge, skills, satisfaction with the educational approach and professional attitudes.

Certainly more use of blended approaches that include digital offers potential to reduce disruption to service delivery from provider absence, reduce the massive amounts possibly wasted through conventional classroom training and harness the power of digital and data analytics for more tailored learning experiences.

6

Mott MacDonald | Mapping and Assessing

Classified as Internal

The following lessons have emerged on the relevance, feasibility and acceptability of technologies commonly used in digital training or data exchange. ➢

Functional requirements include ability to work in a low bandwidth or offline environment, ease of use and low costs

➢

Participants generally prefer to access content on their own devices rather than carry separate devices (although in some settings health providers do not own their own devices)

➢

Mobile phone training through Interactive Voice Response (IVR) can be an effective complement to classroom training for CHWs, even in fragile states

➢

Multi-purpose systems which allow functional integration are far more likely to be used

➢

Support for change management and technological troubleshooting, both for MoH officials and for health workers themselves, is vital to allow transfer of ownership

➢

Building modular content using existing “global good” platforms supports integration, interoperability and sustainability

➢

Open source is desirable but does not mean solutions become “free”.

Better, Fewer Metrics The issue of measurement of performance strategies’ impact is widely seen as an area in need of attention. It is essential for personalised learning, course correction, ensuring strategies can be optimally tailored to context and for ensuring cost-effectiveness of investments. While tailored, personalised learning is seen as priority, the challenge is delivering this at scale. More than ever, pedagogical experts and software developers need to work closely together, to ensure that the power of digital and data analytics can be used to accurately pick up individual weaknesses or difficulties and support tailored learning. This includes greater sensitivity of measurement with feedback loops and remedial action to ensure learning needs are met.

Per diems have become a de facto salary top up in many settings yet simply incentivise attendance at training rather than impacting performance. In the context of low resource settings and constraints on budgets to adequately pay the health workforce, there is certainly an argument for official development assistance (ODA) funds being channelled to frontline health workers if this can leverage quality. Digital technology can be used for virtual, direct observation, as part of more tailored learner-centric learning approaches, as well as for community feedback. Budgets currently spent on training could be partially redeployed to incentivise better quality care, based on the “better, fewer metrics” and people-centred measurement advocated by the Lancet Commission.

Sustainability All the case study countries are embracing digital health in some form, albeit in a more limited way in the most fragile, Somalia and DRC. There are some novel and excellent approaches, though even the most established are still struggling to find a truly sustainable funding model.

Digital learning and performance management need to be seen within the wider lens of digital innovation which faces the same challenges as any new intervention in global health.

Sustainability is as always, the biggest. Ultimately until a major intervention is embedded in a country’s relevant sector plan with funding provided by the government or by individuals it will not be sustainable. In countries such as Ethiopia and Rwanda, where the government takes a strong lead in setting the agenda, providing adequate funding for maintenance etc. and coordinating partners accordingly, country owned sustainable solutions are more possible.

Donors and implementing partners have an important role to play in helping ensure solutions align as far as possible to a coherent national plan - particularly in settings where capacity and digital maturity are less advanced. This means ensuring an aligned approach, investing in

7

Mott MacDonald | Mapping and Assessing

Classified as Internal

interoperable systems and ensuring that new innovations are compatible with the existing digital infrastructure.

Way Forward In taking forward support for strategies which accelerate health provider competency development and performance, Gavi should consider its unique role in supporting solutions to achieve scale and gain traction at country level and globally. The following actions are suggested: 1. Join forces with other multilateral organisations such as WHO, UNICEF, the World Bank and The Global Fund using their platforms to support improved global and in-country coordination of frontline health worker interventions. This could include: o

o o o

Advocating for more internationally recognised, nationally accredited, usercentric learning experiences including via the WHO Academy in collaboration with local ministries, professional councils and associations; Investing in courses for vaccination managers and further language translations of successful courses; Supporting ministries of health in fragile settings establish databases for health provider training to avoid duplication; Running coordinated live simulation exercises with in-country partners e.g. for pandemic preparedness / vaccine roll out.

2. In relation to digital learning and performance solutions, build on what exists and invest in further language translations, local adaptation and contextualisation, supporting interoperability and sustainable funding mechanisms for existing proven technological solutions; in particular, o

o o

Using blended, multi-faceted approaches rather than a single approach – opting for integrated digital solutions which address competency building, performance feedback, and accountability measures such as digital vaccination tracking; Supporting initiatives such as Digital Classroom which can rapidly update, translate and roll out materials Encouraging more partnerships between platforms such as CommCare and Community Health Toolkit in support of interoperability.

3. Ensure rollout of solutions for provider education / performance is country driven and aligned with wider support (through HSS grants) to address health workforce and other health systems constraints. This might include: o

o o

Providing technical assistance to countries to address staff turnover by developing career ladders for frontline health workers, with performance reward and recognition strategies, and investment cases for approaches to reduce wasteful turnover; Ensuring in-service training strategies have MoH buy-in and there is a clear link to career development; investment in a network of local partners for sustainability; Fostering greater links with pre-service training institutions, and providing technical support to digitise elements of pre-service curricula; Consolidating and leveraging more value from existing platforms, for greater economies of scale and to build familiarity with common platforms.

4. Ensure that all innovative/digital investments are aligned to the country digital health plan if it exists, digital development principles and country cMYPs. Consider Gavi becoming a signatory to the digital development principles. Where a digital health plan does not exist, encourage / support the development of one. In terms of aid effectiveness and coordination, ensure investments are aligned with and communicated to other stakeholders/donors; encourage co-financing from country government for

8

Mott MacDonald | Mapping and Assessing

Classified as Internal

implementation of digital technologies, and build up capacities of local Digitech companies through north-south business partnerships; 5. Channel support to district level capacity building for quality improvement initiatives This could include: o

Identifying lagging behind districts and agreeing collaborative quality improvement initiatives - identification of the problem, root cause analysis, action planning, agreement on focused indicators, data collection and rapid feedback cycles (“Roadmap-Lite” approach);

o

Deploying a cadre of technically savvy graduates to be embedded within district teams to support digital elements within above initiatives, particularly troubleshooting use of digital data systems and producing data visualisations; Capacity building of governments for in-house data analytics and sustainable use of data; Supporting district-level management training, to build exposure to less hierarchical, problem-solving / coaching approaches.

o o

6. Seek to measure impact in relation to the original problem that is being addressed using better, fewer, metrics that reflect quality of service delivery. This might include: o

o

Ensuring regular monitoring of a limited number of indicators, rapid feedback cycles and two-way data flow that allows adaptation, layering and continual improvement of interventions; Using mobile phone surveys for collecting rapid feedback from health service users on quality of care, missed opportunities for integrated care etc.

7. Ensure better alignment of funding and incentives and consider innovative payment for performance mechanisms to replace training per diems, based on empirical evidence of service improvement.

8. Allow disruptive bottom up strategies that challenge traditional power dynamics, but explore the feasibility of pushing similar models down to lower tiers where nonEnglish/French speaking participants can engage in their own language. This might include: o

Exploring learning laboratory approaches to distil and continually refine such approaches;

o

Considering a financial mechanism to support “bottom-up” action plans developed through this route via HSS grants, for example micro grants which alumni of such networks could apply for (with involvement of district health teams) for small scale collaborative quality improvement initiatives.

9

Mott MacDonald | Mapping and Assessing

Classified as Internal

1 Introduction Since the early 2000s, there have been large increases in donor financing of human resources for health (HRH), -currently estimated at estimated at $4-6 bn per annum3 - yet much investment by external funders is not coordinated with national strategic plans and there has been little analysis of effects on performance. Too often training is a default reaction to a health system failure – popular because it is comparatively straightforward to deliver, with “numbers trained” an easy metric to report, good for satisfying short term delivery targets of funders.

Although there is little formal evidence there is a common view that a lot of money is wasted on ineffective training. Measurement tools for assessing the effectiveness of training are often weak, relying on tests which measure short term memory of knowledge, or self-reported learning translation. Knowledge has quite a weak correlation with performance (it is necessary but not sufficient). Another part of the problem is fragmentation – in Uganda, 22 different designated organisations responsible for training CHWs were counted. 4 With advances in technology issues of proliferation, fragmentation and lack of alignment of solutions with national plans have come to the fore. Concurrently innovation has rocketed and the multitude of different players has thrown up some novel and excellent approaches, though even the most established are still struggling to find a truly sustainable funding model. The challenge now perhaps is to consolidate the best of these and look for ways to increase value from existing solutions and platforms that exist, leveraging the inherent economies of scale, rather than investing in developing yet more.

COVID-19 has brought about not only new imperatives but also opportunities to make use of new technologies to disrupt the way things are done and accelerate our engagement with technology. The need to limit face-to-face interaction, upskill large numbers quickly and cope with even more pronounced fiscal and economic pressures worldwide speak to the need for more efficient approaches. Many people have engaged more profoundly with new technologies for the first time.

Insights from learning theory and related disciplines like behavioural insights provide a muchenhanced evidence base of what works to improve learning and performance. From this, we understand that content-driven, one-size fits all approaches are only effective for specific situations and need to be complemented by more learner-centred, layered approaches in others.

This report provides a summary of the evidence base around innovative learning and performance management strategies relevant to frontline health workers. We provide an assessment of some notable examples of learning and performance strategies being used in lower- and middle-income countries (LMICs), considering their enabling features as well as the main constraints, keeping in mind a health systems lens. We examine examples of good practice and adaptations which have been made because of COVID-19, and the potential for promising approaches to be scaled up elsewhere.

Background to assignment

Gavi has contracted Mott MacDonald to undertake a mapping and assessment of learning and performance management approaches of frontline health workers. The purpose is to support Gavi’s efforts in transforming learning and performance approaches for frontline healthcare 3 4

Personal communication with Jim Campbell, WHO, 13 January 2020 O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open 2018;8: e021467. doi:10.1136/bmjopen-2017-021467

10

Mott MacDonald | Mapping and Assessing

Classified as Internal

workers in portfolio countries by mapping and assessing existing strategies, identifying gaps and highlighting innovations that can be scaled-up through Gavi’s support.

The output of this work will be applied for two main functions: 1) To inform Gavi’s support to countries during its new 5.0 strategy, in particular building health workforce capacity to address the problem of high unreached (zero dose) children and under-vaccinated populations.

2) By identifying ways to support routine immunisation and rollout of existing vaccines, to prepare health systems for COVAX and other vaccine introduction.

This report presents the outcome from this work. We review the evidence-base on approaches which impact health provider performance, including learning/digital learning strategies, performance supporting strategies and wider health systems levers (Section 2). In Section 3, we discuss implementation experience and some enabling principles which support innovative performance strategies, highlighting some of the platforms and initiatives that apply best practice, including use of appropriate technology, and which could be rolled out at increased scale. Due to a plethora of interventions, there will be many that it is impossible to mention but which are nevertheless making important contributions. Section 4 looks at emerging technologies and future trends, and also reviews experience with adapting to virtual approaches from the education and agriculture sectors. In Section 5 we discuss the results from a deeper dive look at selected countries to illustrate some of the country level issues and explore what approaches have worked in various contexts as well as the path to sustainability. Given the need for better donor alignment and coordination, Section 6 presents a summary of how other funders are engaging with HRH workforce and performance. Sustainability is explored in more detail in Section 7, expanding on themes that emerged from review of interventions and key informant interviews. Finally, Section 8 summarises some of the key strategic opportunities where Gavi could act. A list of key informant interviews (KII) is at annex 1, and a bibliography of key references is at annex 2.

Through our literature and web review and KIIs, we identified prominent examples of innovative solutions to frontline health provider learning and performance. We sought examples which used blended and innovative approaches, including digital, and which had shown signs of adaptation during the COVID pandemic. Solutions were assessed against dimensions of effectiveness, scalability and sustainability. The results of this exercise – with 24 highly ranked solutions - are presented in annex 3 along with a note on scoring methodology at annex 4. An inventory of common global platforms for digital health interventions is an annex 5.

As this is a fast-moving space, and new evidence is emerging all the time, we strongly recommend that Gavi uses these results only indicatively and keeps under review promising solutions for which no data is yet available. Similarly, as interventions do not always translate well to different contexts, we also recommend that solutions being supported for scale up be required to demonstrate alignment to country strategies and plans and government buy-in.

11

Mott MacDonald | Mapping and Assessing

Classified as Internal

2 Evidence on Healthcare Provider Performance 2.1

Healthcare Provider Performance

Healthcare provider performance has been described as a relatively broad construct that encompasses availability, clinical competence, responsiveness (providing patient-centred care), and productivity (or efficiency).5 We know from the various evidence reviews that knowledge is necessary but not sufficient for provider performance and that this is shaped by many more complex factors. A WHO global survey of LMICs on efforts to improve health worker performance found that health worker practices are positively related to health outcomes.

However, health worker performance is influenced by a range of factors including sufficient budgets for training, supervisory systems, staff motivation and retention, supply chains and back-up health facility readiness.

The Lancet Commission on High Quality Health Systems, a major influential body of work premised on the notion that high-quality, people-centred care should be the raison d’être of the health system, asserts that moving to a high-quality health system is primarily a political, not technical, decision. Governments should start, it suggests, by establishing a national quality guarantee for health services, specifying the level of competence and user experience that people can expect, and against which governments would be held to account. It argues that incremental tweaks are insufficient and that a “major reboot” of health systems is needed.

One year on, the Lancet Commission reported that policymakers had agreed that it was time to look beyond marginal improvements and consider major reforms. Universal Health Coverage (UHC) could serve as an entry point to reimagine health systems that are fit for purpose, with strong financing but also new models of governance, provider training, service delivery, and community involvement. The Commission proposes that health systems be judged primarily on impacts, including better health and its equitable distribution; on the confidence of people in their health system; and on processes of care, consisting of competent care and positive user experience.

Provider performance, therefore, cannot be considered in isolation from the wider context of health system reform, the need for better governance and accountability, and a more peoplecentred approach. A more health systems focused approach will also counter the proliferation of many micro-measures which can indeed be deleterious to wider improvements. While accountability starts at the very top, we focus here specifically on health workers and how their performance can be supported by such approaches.

According to behavioural science, we are more likely to practice a desired behaviour if it is easy, if it attracts us, if other people expect us to do it (social norms) and will think poorly of us if we fail, and if it is a “default”, for example more people will donate organs with an “opt out” system than with “opt in”. Contrary to common belief, knowledge and attitude are not good predictors of practice. Despite being aware of guidelines, health providers might not rigorously follow them, possibly with good reason.

One analyst coined the term “mindlines” – which is to say that rather than following technical guidelines literally, what providers do in practice is usually derived from a combination of their knowledge of technical guidelines, their own personal experience implementing those guidelines (which parts had gone well and what not so well), and what their peers do. So 5

WHO. Working together for health. World Health Report 2006. Geneva: World Health Organization, 2006.

12

Mott MacDonald | Mapping and Assessing

Classified as Internal

providers in fact relied on their own personal algorithms in many situations. Similarly, other behaviouralists have used frameworks borrowed from the criminal justice lexicon to think about “means, motive and opportunity” in understanding health providers’ behaviour.

The “means” aspect translates essentially to whether the provider has gained the right competencies, through training, coaching, practice and feedback.

The “opportunity” aspect translate to other health systems readiness factors that need to be in place for a health worker to do his or her job well - whether there are supplies, fuel for outreach vehicles, good data to guide scheduling and so forth.

The “motive” aspect relates more to behaviours and norms. People function within networks and have different influencers in their lives. Considering the individual political economy (or circle of influence) in which frontline health providers operate is extremely important. While an externally driven programme can make an action easy, attractive and even incentivised, there also needs to be a normative expectation of behaviours within the provider’s immediate network. This is why the buy-in from health departments, the ultimate employer of health workers, as well as the involvement of communities in which they live and operate, is so important. The natural desire of health workers to want to do their job well and be recognised for this by their peers can be one of the most powerful enablers to be harnessed.

Review of the most promising examples of scalable innovation suggests factors in success relate to the extent solutions are integrated into existing systems, institutional, behavioural and political contexts and resource environments. This suggests that a broader, multifaceted, health systems approach to performance is needed.

Applying a health systems lens, the starting point should be not whether a particular solution “works”, but rather asking what is the particular challenge, what are the causes, and what can be done about it. This way of thinking would facilitate selection of innovative digital tools only if they are appropriate for identified challenges.6 We consider below the evidence base and examples of various approaches that impact performance, including learning strategies, performance strategies such as collaborative quality improvement and supportive supervision, and wider health systems levers such as payment mechanisms, data for accountability and recognition, and community oversight.

2.2

Evidence from Systematic Reviews

The Health Care Provider Performance Review (HCPPR)7 by Alexander Rowe et al is the most comprehensive systematic review of strategies to improve health-care provider performance in LMICs that has been done. It selected 670 reports from 337 studies of 118 strategies in 64 countries. Its objective was to assess the effectiveness of all strategies to improve health-care provider performance outcomes in LMICs. This section draws heavily on the results of this review and follow up analysis. Overall, this evidence review found that few strategies are clearly effective across the board; results vary widely and there is a great deal of heterogeneity in contexts and implementation quality.

The review found that multifaceted strategies targeting infrastructure, supervision, other management techniques, training and group problem solving tended to have large effects.

Combining training and supervision had larger effects than use of either strategy alone. Group problem solving alone or as part of training showed large improvements in percentage 6

See also WHO Digital Implementation Investment Guide https://www.who.int/publications/i/item/9789240010567

7

Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Alexander K Rowe, Samantha Y Rowe, David H Peters, Kathleen A Holloway*, John Chalker, Dennis Ross-Degnan. Lancet Glob Health 2018; 6: e1163–75. Published Online October 8, 2018

http://dx.doi.org/10.1016/S2214-109X(18)30398-X

13

Mott MacDonald | Mapping and Assessing

Classified as Internal

outcomes. Some specific group problem solving strategies such as “collaborative improvement” work well when a network of facilities come together as a cluster (e.g. facilities in geographical area or of same facility type) and collaborate to solve a problem. However the HCPPR analysis suggests that strategies such as group problem solving might be more effective in areas with higher levels of resources than in low-resource settings, i.e. hospitals in low-income countries and areas in middle-income countries that were not entirely rural. Although reasons for this are not entirely understood, this could possibly relate to a critical mass needed of team members available to collaborate on a QI strategy. The influence of context on strategy effectiveness is unsurprisingly highly significant, hence the importance of considering context and then trialling, monitoring and adapting strategies.

For professional health-care providers (generally, facility-based health workers), the effects were near zero for only providing printed information or job aids or Information and Communication Technology (ICT) alone, suggesting that such approaches work best as a complement to other approaches.

In Bihar, India, a digital Continuum of Care Services (CCS) intervention was implemented by CARE as part of the Ananya program. This involved the provision of mobile-phone-based tools for frontline workers (FLWs) that aimed to increase the coverage and quality of services that FLWs provide and facilitate supervision. The tools combined registration of beneficiaries, scheduling of home visits, and guided protocols along with audio-visual job aids. FLWs received reminders about the timing of home visits and the tool included checklists of information to gather from and provide to beneficiaries during home visits.

A study8 found that FLWs’ understanding of the CCS tool increased but not until the second year of implementation and as a result of an intensive training effort by CARE. Formal training consisted of 16 sessions of approximately 3 hours each held over 8 weeks, as well as informal mentoring involving CARE staff visiting FLWs who were identified as having difficulty. More literate and younger FLWs were significantly more likely to understand the tool. This suggests that hand-holding support needed when introducing new systems needs to be prolonged and not under-estimated.

An important finding was that the level of understanding by FLW supervisors was more limited with about half the supervisors surveyed at endline unable to log into their phone or open the records of home visits conducted by the FLWs they were supervising. FLW reports did not suggest any substantial improvements in supervision of FLWs outside of sub-centre meetings, which was one of the aims of providing the supervisory mobile tool to the ANM. Impact of the assessment across various tasks was mixed, but effect sizes were generally low, in single figures. A plausible explanation is the lack of clear expectation set by supervisors, many of who themselves did not engage with the tool.

The scope of potential ICT approaches is, however, impossibly wide and there has been further work on refining the definition and typologies covered by this (e.g. by WHO) so that these can be assessed in a more nuanced way.

For lay health workers (typically Community Health Workers), training tended to have fairly small average effects. Strategies that included community support plus training were more likely to lead to larger improvements, although the evidence is limited.

Many effect sizes were less than 30 percentage points, which means that even after implementing improvement strategies, important performance gaps will probably remain. The 8

Borkum E, Sivasankaran A, Sridharan S, Rotz D, Sethi S, Manoranjini M, Ramakrishnan L, Rangarajan A. Final Report: Evaluation of the information and communication technology (ICT) continuum of care services (CCS) intervention in Bihar. Princeton, NJ: Mathematica Policy Research; 2015. 82 p. Reference No.: 40195. Contract No.: 4993/22539(18). Supported by Bill and Melinda Gates Foundation.

14

Mott MacDonald | Mapping and Assessing

Classified as Internal

authors recommend the need for continual monitoring of strategies, so that managers can know how well a strategy is working, address gaps (which are to be expected) by modifying or abandoning the strategy or layering on a new one, and continue to monitor and modify as needed.

Overall, the systematic review emphasises the need for researchers to use better methods to study the effectiveness of interventions. Given what is argued above about the need for iterative improvements where approaches can adapt, layer new elements and improve, it is particularly important that any evaluations allow for a dynamic learning approach rather than a rigid intervention design.

2.3

Training attributes associated with effectiveness

Given that important questions remain about how well training works and the best ways to design training, the systematic review authors undertook further secondary analysis to characterize the effectiveness of training strategies and identify attributes associated with improving provider practices in LMICs.9 This analysis found that educational outreach visits at health workers’ sites were more effective than inAnalysis found that educational service training, which was more effective than peeroutreach visits at health workers’ sites to-peer training and self-study. The effect of inwere more effective than in-service service training when some or all training was done training, which was more effective at the health workers’ worksite was 6.0–10.4 %than peer-to-peer training and selfpoints greater than when all training was done offstudy. site. In-service training that incorporated clinical practice tended to be more effective than training without it, by 6.9–7.4 %-points. Interestingly, five eligible studies of training strategies to improve CHW practices found essentially no effect for in-service training and educational outreach visits.

In terms of longevity of training impact, the average effect of in-service training, when done in isolation, declined with time since training, with training effectiveness waning to zero after around 21 months on average. Importantly, when training was combined with supervision, the mean effect did not tend to decrease over time.

Finally, lower baseline performance was associated with greater response to training; for every 10 %-point decrease in baseline performance level, mean in-service training effectiveness was 1.1–1.5 %-points higher.

While these results suggested that certain approaches were more effective, the variability of results and the overall low-quality of evidence suggest that (as the larger HCPPR emphasized) programmes should monitor performance to understand the effect of a given approach in their specific context.

These findings from Rowe et al match findings from an earlier review from 2013 of effective inservice training10 which found that use of multiple techniques that allow for interaction and enable learners to process and apply information work best. Case-based learning, clinical simulations, practice and feedback were identified as effective educational techniques. Didactic techniques that involve passive instruction, such as reading or lecture, were found to have little or no impact on learning outcomes. Repetitive interventions, rather than single interventions, were shown to be superior for learning outcomes. Settings similar to the workplace improved

9

The effectiveness of training strategies to improve health care provider practices in low- and middle-income countries.

Rowe AK, et al. BMJ Global Health 2021;6:e003229. doi:10.1136/bmjgh-2020-003229

10

Bluestone, J., Johnson, P., Fullerton, J. et al. Effective in-service training design and delivery: evidence from an integrative literature review. Hum Resour Health 11, 51 (2013). https://doi.org/10.1186/1478-4491-11-51

15

Mott MacDonald | Mapping and Assessing

Classified as Internal

skill acquisition and performance. This review also found that computer-based learning could be equally or more effective than live instruction and more cost efficient if effective techniques were used.

The above findings are also very much consistent with what is known about the science of how adults learn. In 1984, Knowles devised his 4 principles of andragogy –the method and practice of teaching adult learners – see Figure.

A presentation11 at the Teach to Reach Summit, Seattle, Washington, in November 2015 identified attributes of successful training, some of which particularly relate to these four principles and are highlighted below: • • • •

• •

Training uses stories, case studies, problem-based learning, or simulations Trainees are asked to discuss, debate, collaborate or teach each other Trainers provide informational feedback (i.e., rather than only praise or criticism) – what was particularly good about the demonstration (or what could be improved and how) Trainers help trainees tie the training’s objective to a self-relevant, self-transcendent purpose (e.g., for training on treating an illness, trainers helped trainees understand that improving treatment practices will both make them a better, more respected health worker and save the lives of people in their community) Trainers recognise trainees’ growing competence and help trainees develop selfefficacy Trainers ask trainees to make a plan on how the new knowledge would be put to use.

The plan includes setting goals that are short-terms, specific, and moderately challenging.

Competence can be regarded as a combination of knowledge, skill and attitude which translates to performance at a required standard. It is important to distinguish between the learning required to complete a specific task versus developing competency to perform the full spectrum of a health worker’s or a manager’s role. For example an individual task might be to perform a skin pinch to evaluate dehydration in a child with diarrhoea; but health providers need to be able to carry out the entire set of desired practices needed to evaluate and treat a child with diarrhoea.

In the workplace, on-the-job ‘just in time’ learning can be done quite effectively via a reference card or a video, for example, while developing competency as an immunization manager would be a longer process that would require a different approach. Core competencies as an immunization manager might include not only skills in providing vaccination and managing vaccine logistics but also capacity to use data for local decision-making, to analyze one’s own data, and to apply behavioral and social science thinking to partner with communities and clients.

The ultimate measure of training effectiveness is “learning transfer” – the extent to which learning is translated to improved practices in the workplace. Most interventions, if they

11

Adapted from a presentation, citation: Annie Murphy Paul. "Learning Science." Presentation at the Teach to Reach Summit, Seattle, Washington, November 2, 2015.

16

Mott MacDonald | Mapping and Assessing

Classified as Internal

measure it at all, measure only learners’ perceptions of transfer. A recent review12 attempted to distil validated transfer factors that increase the likelihood of learning transfer, and to make these insights practical for trainers, learning architects, and eLearning developers. This review found that learners who acquire practical skills during training will be more successful in transfer, and that learning skills seems more potent than just learning concepts. Learners are more likely to achieve transfer success if they have early opportunities to apply what they have learned in their work. This is because learners will quickly forget what they have learnt without practice, and can also lose confidence and motivation over time. Measurement of impact is further discussed in section 3.9.

2.4

Evidence on CHW Training

A 2018 systematic review13 of ongoing training for Community Health Workers in low-income and middle-income countries suggested that in-service / refresher training was the most neglected and most variable type of CHW training and highlighted the issues of proliferation of different training providers in this space. Interesting, the review found that the majority of ongoing training was delivered in person, with only four studies reporting the use of mobile technologies in training delivery and the majority taking place face to face.

This review suggested that regular in-service training rather than pre-service training was vital, and that co-designing of training with stakeholders including the intended trainees was important to ensure the training was relevant. At the same time, use of a shared training package could help standardise and quality assure training content and bring efficiencies from economies of scale. WhatsApp groups were found to be used effectively for messaging, support supervision, CPD and team-building.

Positive aspects of such social media included its accessibility at times to suit the user, updates, use of a standard and familiar App rather than a bespoke one, ability to substitute for a great amount of face-to-face contact, and that it was useable on users’ own phones rather than requiring laptops or tablets. Downsides were the cost of data and connectivity limitations. 14 A study of CHW training programmes in sub-Saharan Africa and South Asia (supported by various mobile telecoms companies as well as pharmaceutical companies and the mHealth Alliance) concluded that whether face-to-face or remote, effective teaching should be interactive, not simply didactic/transmission of information; that smartphones could be used effectively as job aids, with practical application while the CHW was actually working, and that feedback loops were vital for useful learning.15 Sending text messages to CHW's phones to improve competency has been one of the most evaluated interventions and shows a point improvement in outcomes ranging from 0%-24% for child health interventions reported by different country studies. 16 Regarding training using traditional face-to-face training, meanwhile, a study of community health volunteers in rural

12

Factors That Support Training Transfer: A Brief Synopsis of the Transfer Research. Will Thalheimer, PhD. Work-Learning Research, Inc. January 2020

13

O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middleincome countries: a systematic scoping review of the literature. BMJ Open 2018;8:e021467. doi:10.1136/bmjopen2017-021467

14

O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open. 2018;8:e021467. doi:10.1136/bmjopen-2017-021467

15

The Current State of CHW Training Programs in Sub-Saharan Africa and South Asia: What We Know, What We Don’t Know, and What We Need to Do. One Million Community Health Workers Campaign & mPowering Frontline Health Workers. July 2014

16

Rowe Alexander et al. Improving health worker performance: an ongoing challenge for meeting the sustainable development goals BMJ 2018; 362 :k2813

17

Mott MacDonald | Mapping and Assessing

Classified as Internal

Uganda found a key requirement was that such training needs to have regular, frequent and local meetings in order for it to be effective.17

2.5

Evidence on Digital Approaches to Supporting Healthcare Providers

A systematic review on the effectiveness of mobile health (mHealth) technologies to train healthcare professionals in developing countries18 was done in 2015. Only seven eligible studies related to medical education in developing countries were identified. Three studies assessed the integration of mHealth solutions into the training of allied healthcare professionals, three assessed resident doctors, and another assessed undergraduate medical students. Six of the seven studies used mobile phones as the intervention tool. The majority of studies indicated that mHealth was a promising tool for education and training of healthcare professionals, although definite conclusions were limited due to the study sizes and quality.

In 2016 Johns Hopkins University were commissioned by the Gates Foundation to undertake a review of Mobile Technology in Support of Frontline Health Workers.19 The report summarised current data from over 140 FHW-supported mHealth projects in developing countries, highlighting the emergent trends and best practices in the use of mobile phones, tablets, and technical platforms by FHWs over the last decade. It detailed the key considerations in choosing the type of phone and platform and associated programmatic costs, presented the evidence on the effectiveness of mobile approaches, and established a framework for systematically deploying such tools. They reported a number of key findings, many of which remain relevant 5 years on: • • •

•

• •

Projects primarily supported CHWs and facility staff through facilitating electronic decision support or data collection activities.

74% of respondents reported that mobile devices were provided to FHWs by the projects Functional requirements for a platform included SMS functionality, ability to work in a low bandwidth or offline environment, ability to create reports and dashboards, and the ability to design workflows. Other broad requirements included open source platforms, ease of use, low costs, interoperability and ease of customization.

Several popular and established platforms (such as CommCare, Oppia Mobile, RapidPro, OpenSRP that are still widely used today) were used to support a range of functions and are interoperable to some degree. Across the board, however, interoperability remained a challenge for mHealth interventions, with little data on the use of standards for data architecture and interoperability for interventions.

Just over 10% of the projects recorded in the database used custom-made or proprietary software.

Evidence suggested that there was potential for cost-savings in the long-run resulting from increasing system efficiency, use of open source technical platforms and content, discounted bulk purchases of equipment and phone services (e.g. minutes, SMS, data plant etc.). They also flagged, however, the lack of consistency in how costs are recorded thereby compromising a rigorous cost effectiveness analysis.

More recently, the prevailing wisdom that there is a lack of evidence on digital learning strategies has been challenged, with over 2,500 RCTs identified in a WHO commissioned

17

Kuule Y, Dobson AE, Woldeyohannes D, Zolfo M, Najjemba R, Edwin BMR, Haven N, Verdonck K, Owiti P and Wilkinson E (2017) Community Health Volunteers in Primary Healthcare in Rural Uganda: Factors Influencing Performance. Front. Public Health 5:62.doi: 10.3389/fpubh.2017.00062

18

The effectiveness of mobile health (mHealth) technologies to train healthcare professionals in developing countries: a review of the literature. O’Donovan J, Bersin A, O’Donovan C. BMJ Innov 2015;1:33–36.

19

Mobile Technology in Support of Frontline Health Workers: A comprehensive overview of the landscape, knowledge gaps and future directions. Smisha Agarwal et al. John Hopkins University. 2016

18

Mott MacDonald | Mapping and Assessing

Classified as Internal

systematic review of digital learning.20 WHO has published a discussion paper “Digital education for building health workforce capacity”21 on which many key stakeholders collaborated. This confirms evidence pointing to digital education being at least as effective as traditional education in improving health professionals’ knowledge, skills, satisfaction with the educational approach and professional attitudes.

This paper suggests that offline digital education may improve educational outcomes such as knowledge of trainees and that significant levels of communication competencies can be delivered via digital education, through online digital education, virtual reality and virtual patients.22 The paper also argues that digital education for health workers can be used as an adjunct to traditional education, usually in a blended learning model. It acknowledges, however, that evidence is often inconclusive, for instance, “it appears that end-to-end digital education is more effective in improving knowledge, whereas blended learning is more effective in improving skills, but this also depends on the learning objectives or assessment methods”.

Evidence suggests, for example, that online digital education already plays a notable role in training medical doctors, significantly improving learning outcomes compared with self-directed or face-to-face learning. In the case of antibiotic management, digital education was shown to improve the prescription behaviours of practising physicians better than those who undertook only traditional learning. Blended learning may be more suitable for health care training, however, which commonly needs to combine practical hands on, skill-based training with knowledge of the theory.

One point of note that the paper makes is that most studies (90%) of digital education (of RCT standard) have targeted doctors and much fewer target allied health professionals/frontline health workers such as nurses or pharmacists. It suggests some balancing needed to target more digital education initiatives at these other cadres.

2.6

Specific digital learning approaches

Learning approaches can be broadly classified into traditional, digital and blended approaches.

Digital approaches can in turn be broken down into off-line digital training (materials pre-loaded onto SD card, USB etc and distributed), on-line training (which can still have offline capability), Massive Open Online courses (MOOCs) and other forms of mobile learning. There are also newer and emerging technologies/approaches including virtual reality, augmented reality and serious games – these are covered in section 2.6.

Certainly, the rapid emergence of new technology along with high ownership of mobile phones and expansion of 3G and 4G connectivity in LMICs paves the way for greater inclusion of digital either as stand-alone or as part of blended learning approaches. Selection of media needs to be compatible with the communications infrastructure and the real costs and ability of the intended trainees to access the training – as a one-way process or interactively.

The ability to update, correct or supplement information which is found to be insufficient or incorrect, is vital. Media such as online courses or apps, which allow rapid revision or update, are thus valuable. This benefit has to be balanced against the barriers posed by the cost and practicalities of network capacity of connectivity, compared to distributing content in advance, in fixed form (e.g. in print, or recorded onto SD card, USB, or downloaded and installed onto devices). The larger the scale of the programme, the greater the importance of getting this balance right. In many cases, a combination of fixed content pre-distributed, complemented by 20

Personal communication with Josip Car, Digital Health Education Collaborative from Imperial in Singapore / WHO Collaborating Centre on Digital Health, 25th January 2021

21

Digital education for building health workforce capacity. Geneva: World Health Organization; 2020. Licence: CC BY-NC-SA 3.0 IGO.

22

Kononowicz AA, Woodham LA, Edelbring S, Stathakarou N, Davies D, Saxena N et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019;21(7):e14676 (http://www.ncbi.nlm.nih.gov/pubmed/31267981, accessed 22 November 2019).

19

Mott MacDonald | Mapping and Assessing

Classified as Internal

real-time or periodic updates, is best. This may be achieved by use of messaging channels such as WhatsApp, by face-to-face training sessions, by broadcast or email updates.

The unit (per learner/trainee) cost of developing and producing courses and training modules, and then of running the training, can vary a great deal according to the sophistication and requirements of the media packages used, the number of trainees, how often it can be repeated, and whether or not the costs of providing hardware or paying for connections has to be covered. These factors are context-specific and need to be considered in programme needs and feasibility analysis, planning and design.

Table 1 presents a summary of Digital Education Modalities along with advantages and constraints making them more or less suitable in different settings.

20

Mott MacDonald | Mapping and Assessing

21

Classified as Internal Table 1. Digital Education Modalities (adapted from Table 3.1 Digital education modalities and definitions in WHO Digital Education for Building Workforce Capacity

Digital education modality

Description

Offline training

An intervention that requires no internet or local area network connection and can be delivered through external media including CD-ROM, external hard disc and USB stick

Online training

An intervention that requires the use of a “transmission control protocol” (TCP) and an “internet protocol” (IP) as a standard for the learning activities; also referred to as “online”, “web-based” or “networked”. Can also have offline applicability. An example is cloud-syncable courses offered offline through platforms such as the Moodle open-source learning management system

Interactive Voice Response (IVR) Audio Job Aides

Can work by a “pull” mechanism, available on demand where users dial a short code to hear message. Or delivered through “push” mechanism where health workers receive calls at scheduled time to listen to short modules.

Decision Support Applications

A form of eLearning delivered in several seconds

Advantages

Constraints

- Does not depend on connectivity - Cost-effective to produce and distribute content, especially if health workers use own devices - Advantage of high quality, pre-prepared content

- More difficult to update content quickly - Expensive if programme needs to supply devices

- Easy to update content - If offline functionality, users can download and interact offline - All online learning is scalable – more users at reducing marginal cost

- High data use requirements - Users care about storage space on their phones - Rarely accessible to those in more remote/ impoverished areas without access to internet or smart devices

- Good ownership of regular mobile phones and reasonable 2G coverage.

- Works on basic as well as smart phones - Calls can be free or subsidised – Telecomm /mobile providers incentivised to participate to increase market share - User friendly translation of clinic or management protocols into advice at point of care - Easily scalable - Easy to update content - Good for settings with high rates of mobile internet penetration

- Users often need credit on their phones to receive calls even when free - Low pick up rates for “push” calls (though higher when supervisors engaged) - Can be language barriers in multilingual countries and regions - Guidance is standardised/binary and might not address the specific problem a provider faces - Lack of interoperability between different apps and systems. Can result in parallel systems increasing burden on healthcare workers.

- Providers often unwilling to log into separate systems

Mott MacDonald | Mapping and Assessing

22

Classified as Internal Serious games and gamification

Serious games are games designed specifically for the “serious” purpose of providing health professional education via a digital. Gamification is “the application of the characteristics and benefits of games to real world processes or problems”.

Massive open online course (MOOC)

An online course that is designed for the participation of large numbers of geographically dispersed students.

Virtual Reality

VR is a technology that allows the user to explore and manipulate computer-generated real or artificial three-dimensional (3D) multimedia sensory environments in real time. It allows for a first-person active learning experience through different levels of immersion.

Virtual patient (VP)

Interactive computer simulations of real-life clinical scenarios for the purpose of medical training, education or assessment

Mobile learning

Learning across multiple contexts, through social and content interactions, using portable, networked devices

(m-learning) e.g.

WhatsApp groups

- Strengthens learning experience by introducing enjoyment, competition and problem-solving - Novel experience increases engagement and attention

- Some games require expensive console / PC equipment - May be less applicable to engage senior staff - Lack of evidence in LMIC settings

- Represent a disruptive vision to get learning out to the masses - Affordable, scalable, and easy to update content - Newer versions emphasise need for MOOC in combination with other learner support

- Have been high drop-out rates - Some MOOCs, especially older versions, were technology driven but weak on pedagogy - Requires access to computing devices

- Time efficient and cost effective in the long run - Realistic, repetitive practice allows for an enhanced learning experience - Potentially replicates positive attributes of training in user’s own work setting and need for incorporation of clinical practice

- High set up costs to roll out equipment and scale up in low- and middle-income settings

- Enables clinicians to continuously practice skills and techniques - Standardised, safe and reduces reliance of patient availability

- Quite complex to design and set up to achieve quality learning.

- Requires local context adaptation and validation

- Rich learning and problem-solving potential through real time, two-way communication - Social media platforms, such as WhatsApp, are commonly used for communication globally - Affordable data bundles increasingly provided by Telecomm/mobile providers specifically for WhatsApp use

- Users need to use their own data bundle which can result in increased personal costs to the user - Numerous messages could evoke user fatigue - Encroaches on personal life and could impact wellbeing - Only available on smartphones

Mott MacDonald | Mapping and Assessing

23

Classified as Internal

2.7

Performance Theory

For decades, researchers have developed and built on models which seek to explain human performance and how this is best shaped. The Behavior Engineering Model (BEM) developed

by Thomas Gilbert23 is fundamental to many later models and provides a way of systematically identifying barriers to individual and organisational performance.24 Roger Chevalier updated and adapted this model using language commonly employed in describing worker performance.

Environment

1.Information

2.Resources

3.Incentives

Individual

6.Knowledge

5.Capacity

4.Motives

As with the original BEM, it focuses on the distinction between environmental and individual factors, with environmental factors as the starting point because, in the words of Geary Rummler and Alan Brache, “If you pit a good performer against a bad system, the system will win almost every time”.25 In the article “Training Alone is not Enough”, 26 the authors present findings from IntraHealth’s research in Human Performance Technology (HPT) on the factors influencing family planning provider performance. The study drew on earlier human performance models (developed by Rummler & Brache in 1993; and by Stolovitch & Keeps in 1999) that assumed that clear expectations, timely performance feedback, adequate environment, incentives (and consequences), and skills and knowledge are required for a worker to perform well. The study explored how each of these conditions affected healthcare provider performance in four countries.

This study showed significant association between the performance of workers and conditions in the workers’ environment. When viewed across all countries, non-monetary incentives were the most powerful “predictor’ of performance when compared with other factors while knowledge and skills were found to be the least influential factor overall.

These findings are highly significant given the dominance of training as a relatively expensive solution and the concurrent under emphasis on good management practices - which include performance reviews to set clear expectations, feedback and verbal recognition - as well as approaches which s increased respect from both the community and the supervisor will have zero or small financial implications but a potentially large impact.

The following sections look at some of the practical ways a performance culture can be established.

2.8

Collaborative Quality Improvement

Among performance management strategies, collaborative quality improvement (CQIs) have been used to improve health care for several decades. CQIs involve the use of healthcare teams from different sites to improve performance on a specific topic by collecting data and testing ideas with improvement cycles which can be supported by coaching and learning sessions. The district level of the health system is well positioned to facilitate systematic group learning among facilities of similar types and across tiers of the health system. District-led area-

23

Human Competence: Engineering Worthy Performance. Thomas Gilbert, 1978

24

Updating the Behavior Engineering Model. Roger Chevalier, CPT Performance Improvement, v42 n5 May-Jun 2003. http://www.ispi.org/publications/pitocs/piMay2003.htm/

25

Ibid citing Rummler & Brache, 1995

26

Training Alone Is Not Enough.- Factors that Influence the Performance of Healthcare Providers in Armenia, Bangladesh, Bolivia, and Nigeria. Lauren Crigler et al. Performance Improvement Quarterly, 19(1) PP 99-116

Mott MacDonald | Mapping and Assessing

Classified as Internal

based learning and planning bring together providers and administrators responsible for a catchment area to solve clinical and system problems, harmonize approaches, maximize often limited resources and create better communication and referral between facilities. 27 Evidence shows that CQI when supplemented with health worker training result in positive patient outcomes for formal health providers. There is less known about the positive health impact from such approaches in the context of lay health workers. 28 The following example is drawn from a successful trial of a CQI in Niger and Mali. 29

Example 1: CQI to address Post-partum Care in Niger and Mali A large-scale programme in Niger and Mali sought to improve postpartum care for mothers and newborns. The approach involved identification of the root causes of failures to complete all steps in the active management of 3rd stage of labour. Facilities collaborated in identifying and trialling solutions to the specific micro-issues identified, for example the fact that although oxytocin injection was indicated within 1 minute of birth, this rarely happened in time. New solutions were trialled, for example keeping a pre-prepared loaded syringe in an ice-bucket next to the delivery bed.

Importantly, data was collected and used by the facilities themselves at very regular intervals on where measures introduced were having effects. As a result of this improvement effort, local ownership and shared learning to accelerate implementation of best practices, 78 facilities demonstrated rapid improvement incompliance with standards for post-partum haemorrhage prevention and essential newborn care as well as a reduction in estimated postpartum haemorrhage. The case study found that this approach yields rapid results and can be efficiently spread to improve care in low-resource settings.

Similarly, a research study in South Africa concluded that an approach of continuous quality improvement (CQI) based on continuous mentoring of CHWs by their supervisors was found to improve the CHWs’ delivery of services to pregnant women and mothers in both quantitative and qualitative terms. The CQI followed an initial two-week face-to-face training in WHO’s Community Case Management. The study concluded that training using low-tech, face-to-face mentoring, with printed support materials, could be effective, on the basis that it was carried out locally and with frequent, regular mentor-trainee interactions built into the CHW’s annual calendar.30

2.9

Supportive Supervision

The effect of supportive supervision strategies on health worker practices in LMICs is highlighted by another review which identified attributes associated with impactful supervision. 31 Provision of supervision to supervisors and supervisors’ engagement in problem solving with healthcare providers, were two features strongly associated with a positive effect on provider 27

Garcia-Elorrio E, Rowe SY, Teijeiro ME, Ciapponi A, Rowe AK (2019) The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: A systematic review and meta-analysis. PLoS ONE 14(10): e0221919. https://doi.org/10.1371/journal.pone.0221919

28

Ibid

29

Improving postpartum care for mothers and newborns in Niger and Mali: a case study of an integrated maternal and newborn improvement programme. M Boucar et al. USAID Applying Science to Strengthen Improve Systems University Research Co., LLC 18 September 2014

30

Horwood et al. Human Resources for Health (2017) 15:39. DOI 10.1186/s12960-017-0210-7

31

Samantha Y. Rowe, PhD*; Dennis Ross-Degnan, ScD; David H. Peters, DrPH; Kathleen A. Holloway, PhD; Alexander K. Rowe, MD. The effectiveness of supervision strategies to improve health care provider practices in low- and middle-income countries, 2020 (draft, under publication)

24

Mott MacDonald | Mapping and Assessing

Classified as Internal

practices, from an analysis of 81 studies across 36 countries. Training of supervisors, provision of supervisory checklists, and frequency of supervisory visits were not meaningfully associated with health worker practices.

Previously, hierarchical models of supervision emphasising inspection and control have been promoted, but more collaborative supervisory strategies are now widely advocated. Such strategies might typically involve record reviews, observations, performance monitoring, constructive feedback, provider participation, coaching, problem solving, and focused education.

A study from 201632 looked back over nationally representative surveys of health systems in seven countries in sub-Saharan Africa and pooled these with clinical observations to examine the association of in-service training and supervision, both popular strategies, with provider quality. The results showed that observed quality of care was poor, and associations between quality and in-service training and supervision were modest— although there were marked variations between countries. However, at most, improvements related to interventions were equivalent to 2 additional provider actions out of the 18–40 actions expected per visit.

This study also found that effective supervision must include problem solving in the context of an ongoing supervisory relationship.33 The authors suggest promising strategies such as addressing supply shortages in facilities in conjunction with coaching providers; more effective use of data feedback for health workers combined with action plans detailing appropriate responses to challenges in providing patient care, and engagement and capacity building of health system managers to identify poor quality and take actions to address it.

The role of mobile phones to strengthen supportive supervision for CHWs was studied in Kenya.

A WhatsApp group to facilitate instant messaging was created for CHWs and their supervisors to ‘support supervision, professional development, and team building’. Importantly, the authors of this study reported on the quality assurance and information exchange, which the system facilitated, and on the supportive environment fostered by the use of the technology.34

2.10

Payment for performance

Payment for Performance (P4P) - There is a considerable literature on this but much is dated ten to fifteen years ago and interest seems to have peaked. There is considerable evidence from the hospital sector in all countries that provider payment mechanisms can affect performance positively or negatively. In the UK the capitation system for primary care is adjusted to reward key targets including immunisation coverage and appears to be effective in that targets are met. However there has not been a control study. In primary care in LMICs there is no clear evidence that P4P works. One reason is there is not a standard blueprint for P4P schemes being used in LMICs. A recent scoping review identified 41 P4P schemes in 29 LMICs. This variation may well reflect the fact that P4P schemes have been tailored to meet certain policy objectives and respond to different conditions on the ground. The review could not identify common features that would affect results.

A useful review concludes the evidence on P4P in LMICs is still in its infancy, both in terms of evidence of impact (especially as far as health outcomes are concerned), and in terms of the attention to potential unintended consequences.

32

Training And Supervision Did Not Meaningfully Improve Quality Of Care For Pregnant Women Or Sick Children In Sub-Saharan Africa | Health Affairs. Hannah H. Leslie Anna Gage Humphreys Nsona Lisa R. Hirschhorn Margaret E. Kruk: September 2016 https://DOI.ORG/10.1377/HLTHAFF.2016.0261

33

Measuring the quality of supervisor–provider interactions in health care facilities in Zimbabwe. P Tavrow, YMI Kim, L Malianga - International Journal for Quality …, 2002 - academic.oup.com

34

Henry JV, Winters N, Lakati A, et al. Enhancing the supervision of community health workers with whatsapp mobile messaging: qualitative findings from 2 low-resource settings in Kenya. Glob Health Sci Pract 2016;4:311–25.

25

Mott MacDonald | Mapping and Assessing

Classified as Internal

One concern with P4P is the potential for gaming: the data used to measure performance might be manipulated by the service provider to inflate reported performance for example for child immunisation, and it can be difficult to verify that the immunisation took place. The ideal response to gaming is for the funder to base the P4P scheme on information which is outside of the control of the delivery organisation and which is easily measurable, so that there is little scope for misreporting. Vaccinator tracking systems using GIS technology and photo upload are proving useful for verification of outreach work.

A second concern is cherry picking. If the cost of providing an incentivised service differs across patients, and differences in costs are not reflected in the P4P payment, the delivery organisation may have a financial incentive to select patients with low cost and avoid patients whose costs are above the tariff e.g., hard to reach children and mothers. Solutions which mitigate this include reimbursing against mileage travelled with GIS verification, so health workers visiting more distant locations for immunisation outreach, for example, are properly compensated.

Overall introducing P4P schemes needs to be approached with considerable caution with the need to ensure the country health system has the resources to design, implement and monitor it effectively. The costs of independent verification can also be disproportionately high, which is why integrated platforms that have a built-in verification function are valuable.

2.11

Data for Accountability and Recognition

Making data public has an impact on performance as this can foster healthy competition, and acts as a disincentive for poor performance that is within the control of the health worker.

Frontline health workers often work among rural populations, with only sporadic contact with supervisory staff. Web-based dashboards allow supervisors to track the performance of community health workers individually or at the district/regional/national level, either by noting the volume of digital productivity or by real-time GPS tracking of workers as they perform their field activities. This enables supportive supervision to those workers who may be lagging in their performance, while also enabling the recognition and reward of exceptional field staff. These approaches are embedded within a number of mHealth service packages.

In Pakistan, several different but similar approaches for digital vaccination tracking have been pursued: Example 2: EPI Program, Punjab Information Technology Board E-Vaccs digital application scaled out across 3 of 4 provinces of Pakistan designed by the Expanded Program of Immunisation Program in partnership with inhouse IT Punjab Information Technology Board. It features a smartphone application provided to vaccinators which is used to monitor vaccinator attendance, outreach visits through geographical geo-tagging, information feeds on immunisation volume used to construct an e-vaccination card and SMS reminder to children of defaulting parents. Realtime records are streamed through a central database into digital dashboards that are used by the EPI program and senior leadership to review monthly program performance. Evaccs use for immunisation monitoring led to rapid increase in immunisation coverage in Punjab province, attributed to strong government ownership, leadership engagement with immunisation performance monitoring and competition between districts for performance recognition. E-Vaccs has relied mainly on vertical implementation to achieve results and does not have supportive supervision elements, hence with leadership changes in recent years it has faced compliance issue from vaccinators in districts with weak governance.

26

Mott MacDonald | Mapping and Assessing

Classified as Internal

Zindagi Mehfooz is a somewhat similar phone-based, electronic immunisation registry across all 29 districts of Sindh which records and analyses programme data in real time to produce a variety of performance-based immunisation reports, while Teeku is another (see Example 3).

Example 3: Teeku District pilot, Tando Muhammad Khan, Sindh, Pakistan A successful multi-faceted and integrated intervention which included digital data for accountability, supportive supervision and payment for performance was the Teeku pilot in Sindh, funded by Gavi and carried out by Aga Khan University with Department of Health Sindh. This included i) Immunization registry: child and household registration with timelines of scheduled dosage by child and village, defaulter alerts and chat support; ii) GIS tracking of vaccinator visits; iii) Micro-planning targets linked to refreshed annual plan, monthly recognition at district immunisation meetings; iv) Pay for performance: vaccinators were provided fuel allowance based on verified visits. Vaccinators reported that the app was user friendly and helped them to track defaulters. Working vigilantly and ‘honestly’ under their supervisors was reported as a new and enjoyable experience. Photo verification of child vaccination was considered to be the best feature, while a video feature helped educate communities.

District managers reported that the App helped in improving vaccinator compliance and that monthly performance review meetings occurred regularly. While the Micro-plan target setting and fuel money provision linked to App helped chase coverage rates, they reported that notice/ salary suspension to non-performing vaccinators did not always help. Although not scaled further, lessons are being applied to 2 more districts in KP and GB regions.

The concept of SMART regulations involves introducing softer regulations, especially selfregulations, alongside traditional sanctions-based regulations. It works of the premise of coregulation whereby regulator, health providers and a range of other stakeholders such as accreditation institutes, insurance schemes and consumers themselves all co-produce desired behaviour in health care providers through a range of regulatory instruments and broadening of regularly responsibility.

In South Africa, the Office of Standards Compliance (OHSC) launched a patient-centred accountability service called ‘Rate my Clinic’. This was a cell phone-based patient rating system developed to score clinics based on user visit experience. Clinic scoring was done on a range of dimensions including staff attitude, infection prevention, cleanliness, waiting times and drug availability. It served as an early warning system for regulators and helped to effectively apply limited inspectorate resources to low scoring clinics. It also strengthened the interface between quality of care and public accountability. This was piloted successfully in one state before being scaled up across South Africa.

2.12

CHW Performance and Community Involvement

More focused evidence on lay workers is available from a systematic review of interventions to improve CHW performance.35 This found that positive changes in community behavioural outcomes and improved use of services can result from CHW interventions if upstream completion of prescribed activities and downstream adherence to protocols are the focus of interventions. Supervision, incentives and equipment/ commodity support are key parameters that influence CHW performance. Incentives work well when CHW have a single repetitive task but less well when they are expected to multi-task.

35

Ballard m, Montgomery P 2016. BMJ

27

Mott MacDonald | Mapping and Assessing

Classified as Internal

A qualitative review of factors influencing CHW performance proposes that CHW performance is affected by hardware factors such as logistics, supply, training, supervision, accountability mechanisms and communication support, as well as software factors such as relationships, power, values, norms and interests related to community, CHW program management and other healthcare workers.36 Competency development in soft skills such as communication, confidentiality issues and handling community relations is important and relatively over-looked as compared to hard skills of service delivery. Similarly, the role of non-financial incentives is important for CHW delivery and can take the form of transport support (e.g. bicycles), recognition (e.g. CHW days), preferential access to certain health or educational services etc.

In 2018, WHO, UNICEF and The Lancet produced the first evidence-based global guidelines for health policy and system support to optimise community health worker programmes. According to this document, there was evidence that broad strategies (e.g. competency-based education, supportive supervision, and payment) are effective. However typically evidence was not sufficiently granular or too context-specific to allow recommendations on specific interventions for example which supervision strategies or which bundle of financial and non-financial incentives were most effective. But involvement of the community in selecting CHWs, observation of service delivery, performance data and community feedback were all found to be effective. The guideline recommended remunerating practising CHWs with a financial package commensurate to the job demands, complexity, number of hours, training, and roles that they undertake, and recommended not paying CHWs exclusively or predominantly according to performance-based incentives.

The guideline also suggests consideration be given not only to traditional performance measures, but also to basic labour rights that include safe and decent working conditions and freedom from all kinds of discrimination, coercion, and violence. Some of these aspects are of particular concern and relevance in both acute crises and chronic complex emergencies, as there is a growing body of evidence that CHWs have a strong potential in mitigating their negative health impact.

36

Kok, M.C., Broerse, J.E.W., Theobald, S. et al. Performance of community health workers: situating their intermediary position within complex adaptive health systems. Hum Resour Health 15, 59 (2017).

28

Mott MacDonald | Mapping and Assessing

Classified as Internal

3 Enabling Principles for Innovative Performance Strategies This section examines some key principles associated with success of performance strategies derived from the evidence base, and provides examples of these principles being applied in practice as well as explanations for why best practice is not always followed.

Some important principles, many somewhat overlapping, are as follows: ➢ ➢ ➢

➢

➢ ➢ ➢

➢ ➢

Using a blended learning approach, where components can be layered and tailored to requirements, generally works best Using different media and formats tailored to context increases success of training Co-designing with health care providers themselves allows training to focus on practical issues rather than theory, thus becoming more relevant and immediately transferable to health providers’ practice Including problem-solving approaches and collaborative quality improvement addresses specific identified challenges rather than relying on approaches which deliver standardised content Group dynamics, peer support, buddies, and communities of practice are an important part of skill-building and norm setting Ensuring co-production with government is more likely to lead to ownership and government buy-in and sustainability Linking training with accreditation as part of a lifelong learning approach increases uptake/completion rates and ensures better alignment with national health workforce strategies The feasibility of technology must be considered though digital approaches add value in fragile settings Better measurement approaches are needed to support tailored learning

These principles are expanded and illustrated below.

3.1

Using blended and layered approaches

Blended learning is very often the preferred approach and digital interventions that have been used conjunctly with other learning strategies have been more successful than a sole reliance on digitalization. A blended approach makes best use of remote learning through technology, in combination with the richness and close interaction of face-to-face meetings. However, face-toface training may itself be ineffective if not well prepared and not available at the time and place it is needed, or if it is simply transmission of information. It can also be expensive, difficult to achieve to schedule, of low or uncontrolled quality, and disruptive to other commitments.

The remotely-accessed components are of most value for quality of carefully pre-prepared, expert content and learning activities; access at times and in the blocks which are needed by the user at any stage; and for opportunities for recap, for wider peer group interaction and for frequent update.

Dependent on available expertise and budget, interactive, self-check activities and learning games can very usefully be developed and built into the web-based material. Broadcasts on TV or radio, email correspondence, closed chat groups, bulk or individual SMS messaging, individual or group voice calls, printed self-access material can all be useful. Preferably, centrally produced content may be supported and mediated with local practice or workplace application, mentor, tutor or peer group support.

29

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 4: RAPID IMMUNIZATION SKILL ENHANCEMENT (RISE) by JSI & Ministry of Health & Family Welfare, India Rapid Immunization Skill Enhancement (RISE) is a blended package to strengthen the ongoing training of frontline health workers engaged in routine immunization in India with an alternate capacity building model during COVID-19. This user centric model leverages the high technology (mobile phone, internet) penetration among the health workforce and effectively uses a digital training content supported by a face-to-face component of mentoring by supervisors. The digital part is primarily selflearning through an objective focused content based on adult learning principles that engages the learners through a variety of audio, animated visuals, reading and interactions like learning games and quizzes. It also has a provision of objectively assessing the learning and certification to make it more attractive to the learners. An open-source Learning Management System (LMS) provides a realtime monitoring dashboard which enables problem-solving and mentoring by supervisors. The intervention has been rolled out in 5 states of India, is available in 4 regional languages, and has 3,000 health providers to date. Users have fed back that the mobile App could be more user-friendly so that it can be easily operated by all health care workers and that it should be updated frequently as the guidelines in immunisation program changes very often.

Example 5: Community Health Academy: Liberia by Last Mile Health The Community Health Academy partnership was set up in 2017 to deliver educational text, video, and quiz content to first level health workers in Liberia through an integrated training/ support application. 100% of government FLHWs were equipped with digital aids and assessments indicate increase in prenatal visits and correct diagnosis/ referral of malaria cases by health workers.

Liberia’s limited telecommunications infrastructure—compounded by dense tropical rainforest and frequent rain—means connectivity is often sporadic and unreliable. To address these challenges, the Ministry of Health distributed new curriculum using Bluetooth capabilities. The Liberian experience has been drawn on by Harvard University to develop a global blended learning curriculum for CHWs. This is designed to be easily adapted to local contexts and offered as a public good for ministries of health worldwide to adapt, aligned to WHO guidelines, protocols and global standards for quality.

Blended and layered approaches are inherently more complex than simpler, single modality interventions and complexity makes rolling out at scale more difficult. “Cascade approaches” to training are common, where master trainers train others to deliver the training at lower tiers of the system. While this system has advantages of greater coverage at lower cost, quality can be dissipated, and there might be more tendency to stick with a training scheme that people are familiar with.

In this situation, however, a mix of digital and face to face delivery would often be optimal, as digital delivery would allow more standardised, quality-assured delivery, while involving actors from different tiers of the system to deliver training can increase their ownership of the process.

When designing brand new training schemes, it is easier to incorporate digital as part of a blended approach from the start.

Example 6: Surge vaccinators, UK In the UK, the National Health Service (NHS) has brought in St John Ambulance and the Royal Voluntary Service to coordinate the training and deployment of thousands of volunteers to help ensure the safe and smooth running of vaccination services. Volunteers are first screened via on-line application, identity, and police record check and by an on-line interview. Volunteers are trained using a blended learning approach. They first complete on-line training, which can be completed in around 3 days, and need to pass each module before proceeding to the next. Training then culminates in a one-day face to face classroom training, where participants undertake role plays to practice various techniques under different scenarios. They practice giving an injection, injecting water filled syringes into silicon bags strapped onto the arms of fellow volunteers.

30

Mott MacDonald | Mapping and Assessing

Classified as Internal

3.2

Using different media and formats tailored to context

Remote technologies themselves are in general most effective and reliable when used in combinations, rather than solely relying on a single medium in isolation. Media choice may include web-based learning material in audio, video, text and visual graphics.

Films can be used within traditional and digital training settings, and are very effective to create an emotional engagement with the subject matter which supports retention of the message and internalisation of the learning. Films are often valued by communities in unstable and volatile countries, providing women and their families with accessible, relevant and potentially lifesaving knowledge. Films made by Medical Aid films in South Sudan to increase awareness amongst women of the benefits of proper nutrition and correct pre and postnatal care were very popular.

Short films and even text messages to health workers can also be used to reinforce health workers’ credibility and the validity of their advice to communities.

Focus Group participant from Moyo, Uganda ITPp Malaria Trial37

Example 7: Medical Aid Films Medical Aid Films has produced over 300 films which are viewed 500 million times per year. Their films have enjoyed success in low resource settings, used for example in Somaliland as part of elearning for the Nurse and Midwife Association to show techniques to be practiced in parallel in the classroom such as resuscitation on a model baby. Community workers from partner Africa Education Trust report that the films are encouraging positive changes in the health practices of local women and girls. The films are also popular amongst men, with some men and adolescent boys even taking part in subsequent discussion sessions, and also used in facility waiting rooms to encourage impromptu discussions. An excellent film has been produced on vaccine hesitancy, with support from the London School of Hygiene and Tropical Medicine. A key enabler is the ability to translate content to local languages and mix in local footage along with standardised content.

Example 8: eTraining of Nurses for Competency Development: Mozambique Tablet based training based on video modules using scripts and films, followed by skills-based questions was provided to nurses for improving competency to provide lifelong ART therapy to HIV positive pregnant women. Nurses found e-training easier to understand and more engaging than usual face to face paper-based learning, and resulted in increased knowledge competency. A key enabler was appropriate local content of videos and engaging adaptation, helped by formative assessment for design and use of local theatre groups for adaptation.

37

Rassi C, Gore-Langton GR, Gigudu Walimba B, Strachan CE, King R, Basharat S et al (2018) Improving health worker performance through text messaging: A mixed methods evaluation of a pilot intervention designed to increase coverage of intermittent preventive treatment of malaria in pregnancy in West Nile, Uganda. PLoS ONE 13(9): e0203554.

31

Mott MacDonald | Mapping and Assessing

Classified as Internal

3.3

Designing with user needs in mind

It is noteworthy that in a review of ongoing training of CHWs in LMICs, only 5 out of 35 studies reviewed documented seeking input from CHWs as part of qualitative research before design. A major reason for dissatisfaction with much conventional training is inadequately qualified trainers who are unfamiliar with providers’ working environment and ground realities. Given the lack of documented participant input and feedback in terms of programme design, delivery and evaluation, this review suggested greater use of participatory action research (PAR) – i.e. working ‘with’ end-users in a collaborative effort rather than ‘for’ or ‘on’ them.38 There is of course a natural tension between the need not to “reinvent the wheel” and the need to design specific to the user’s context. There might well be existing products that can be adapted, but the real need for adaptation might not be obvious until the tool is properly fieldtested. That is when an organisational inertia to making changes and adapting can set in, particularly when programmes are up against delivery targets. While the phrase adaptive management is much talked about, what it really entails in practice is perhaps under-estimated.

In response to demand for remote training on COVID-19, UNICEF were pushed into a new space and collaborated with the Digital Classroom initiative to adapt their content into a training module for FLWs, initially in Liberia, Chad, Togo and DRC, and now expanding to 8 further countries. The intervention was tested in a pilot with 30 CHWs in Liberia and content was offered across three platforms: regular SMS, Moodle and the Internet of Good Things (IOGT).

The preferred platform was SMS due to CHWs’ existing ease and familiarity with this, with Moodle coming second, which had the advantage of allowing multi-media content.

Example 9: Immunization Academy Watch IAW is a global initiative providing rapid access to short, “just in time” practical videos for task-based immunisation training and delivery in the field. It uses online videos and resource documents accessed through their website as well as an app for smart phones. It covers a wide range of immunisation topics as well as new content on working safely during the COVID-19 pandemic, and has a WhatsApp group of users. Content was developed through in-depth consultation with user groups in Tanzania and combined with existing immunisation “best practices” from WHO. Effectiveness is measured through knowledge increase measured by quizzes, and by the growing number of users, particularly in countries like Nigeria, India and Pakistan. There has not, as yet, been any attempt to assess providers’ competence to carry out the task being trained following exposure to the videos. This initiative is sponsored by a grant from the Bill & Melinda Gates Foundation and led by learning experts at Bull City Learning. For now the only model is donor funding as user populations cannot fund it directly, although a country subscription model for future would provide a more sustainable funding source.

38

O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middleincome countries: a systematic scoping review of the literature. BMJ Open 2018;8:e021467. doi:10.1136/bmjopen2017-021467

32

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 10: Village Reach, with Praekelt and Viamo The partners provide remote, phone-based training to more than 100,000 health workers through a Gavi INFUSE partnership. Praekelt combines two proven solutions together: leveraging the established MomConnect platform and combining with health Hotlines, using WhatsApp for one-toone engagement, targeting both health users and health workers. For health messaging, they offer flexibility so users can choose the best method to reach (whether via smart phone, IVR, or the ability to talk to a person).

Connectivity is an issue and some health workers formed their own groups to listen together to get around this problem (although this made it difficult to count how many had done the training). A module on cold chain maintenance involved a video (using the ECHO platform) but ran into issues with bandwidth. Low pick up rates were an issue, and although free of cost, IVR calls might not connect if a participant has no phone credit. Mobile network operators are providing competitive data bundles for people to engage through WhatsApp. Praekelt has been able to lean on mobile providers to secure good rates although said it was difficult to reach an agreement.

Providers were reported to miss in person interactions, while an important missing link for training was follow up supervision. Remote training is a good way to reinforce simple messages as part of a ‘package’.

3.4

Incorporating problem-solving approaches

A common theme from the evidence is that to be effective, training and supervision need to address specific identified challenges rather than relying on standardised, content-driven approaches. Problem solving has been shown to increase the effectiveness of training and supportive supervision.

WhatsApp has taken off worldwide partly because it is so accessible – mobile operators are now providing competitive data bundles for people so they can engage through WhatsApp. Another important reason for WhatsApp’s popularity is that it allows two-way, real-time communication and short concise answers to day to day issues that arise.

There is currently great interest in the potential of more democratic “bottom up” approaches to sidestep traditional power hierarchies and vested interests in the traditional “training economy”.

So-called Distributed Networks can crowd source solutions to problems by seeking help from peers facing similar challenges, even if working in different locations. Such approaches seek to empower health workers to apply their local knowledge to solve problems.

The Geneva Learning Foundation (TGLF) is the best example of such an approach in the immunisation context (see example 11). Their aim is to contribute to transforming learning, leadership, and training to strengthen immunisation programmes. The approach draws on George Siemens’ Connectivism learning theory which emphasises the importance of networks, self-organization (the spontaneous formation of well-organized structures, patterns, or behaviours), and lessons from chaos theory about complex patterns and effects of small changes in initial conditions that are important properties of learning and decision-making.

TGLF’s approach plays on the moral dimension and reinforcement of positive norms – connecting health providers with other providers who are motivated to do their jobs well and achieve results. Their model pays no per diems or other incentives and expected rewards are derived from doing a meaningful, valuable job well and achieving better results.

There is potential for organisations such as Gavi to explore how learning from this newer innovative model can support its growth, and how action plans identified through selfestablished, voluntary networks can feed into health systems strengthening plans supported at country level, especially where challenges come down to lack of resources for implementation.

33

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 11: Geneva Learning Foundation, Teach to Reach Accelerator Conference.

TGLF’s approach aims to make peer monitoring and support into a scalable system. A Teach to Reach Accelerator Conference was held in January 2021, aimed at country-based participants from its network of over 25,000 people. The approach uses a collaborative problem-solving approach to immunisation challenges which values the inherent knowledge, expertise and experience of health workers about their own working context. Participants prepare action plans, which are peer-reviewed, to deliver on actions in line with country plans. 549 Teach to Reach Alumni collaborated in an intensive three-day Immunisation Training Challenge Hackathon (ITCH) to problem-solve their immunisation training challenges in October 2020. Over 4000 immunisation professionals have joined the Scholar COVID-19 Peer Hub. In July 2020, 500 meetings were held by scholars in countries with no travel or per diems involved, which is indicative of the power of distributed networks as a force for change. Online social networking enables communities of practice that offer potential to bring together temporally and geographically dispersed actors to work towards a common purpose. There is however a risk of lack of coordination and alignment with government plans and specific country performance objectives.

3.5

Communities of practice and peer support

Communities of practice provide a knowledge-sharing platform for peer learning. Virtual communities of practice can be created for case-based learning through use of tele-mentoring, case studies and guides, and multi-directional learning communication. The success of virtual communities of practice is dependent on basic infrastructure such as electricity and internet bandwidth, use of cloud-based videoconferencing platforms through a webcam-enabled computer, tablet, or smartphone, contextual adaptation of case material and familiarity/ comfort of local users and local experts to participate.

Virtual communities of practice such as those that use the Project ECHO model (Example 12) go beyond MOOC-style virtual classrooms by developing knowledge networks that promote real-time multidirectional learning and teaching, with a strong emphasis on peer-to-peer sharing and learning. Knowledge delivery and mentoring are combined with opportunities for live discussion and problem solving in situations where meeting in person is impractical and costprohibitive.

Challenges to the implementation of virtual communities of practice include the need for clinical and public health experts and participants to have protected time to participate, and adapting a model of learning that is non-hierarchical, participatory, and dynamic in contexts where such an approach might be unfamiliar.

Example 12: Extension for Community Healthcare Outcomes (ECHO) Project ECHO is not directed specifically towards frontline workers but is increasingly being adapted to support community-health-worker initiatives. The ECHO Model can be defined as tele-mentoring, a guided practice model where participants are part of a knowledge-sharing platform through virtual communities of practice and case-based learning. It uses multipoint video conferencing that connects specialist teams at local, regional, and international academic medical centres and centres of excellence with primary care teams and community health workers in rural and underserved locales.

The ECHO Institute works with partners to support a global COVID-19 response focusing on IPC, care and management of patients, as well as other diseases and health initiatives.

Virtual communities of practice such as Project ECHO goes beyond virtual classrooms by developing knowledge networks that promote real-time multidirectional learning and teaching, with a strong emphasis on peer-to peer sharing and learning. Knowledge delivery and mentoring are combined with opportunities for live discussion and problem solving in situations where meeting in person is impractical and cost-prohibitive. The success of virtual communities of practice is dependent on basic infrastructure such as electricity and internet bandwidth. However, cloud-based videoconferencing platforms are now accessible to nearly anyone with a webcam-enabled computer, tablet, or smartphone.

34

Mott MacDonald | Mapping and Assessing

Classified as Internal

3.6

Co-production and government buy-in

Ensuring government ownership is a prerequisite to sustainability but can be challenging because of a lack of capacity within ministries, especially in digital technology. Constraints on government spending budgets also hinders efforts to support introduction of new systems while turnover within government departments imparts more challenge.

Decentralised systems (or dysfunctional centralised system) can make obtaining buy-in tricky, as a ministry of health at central level might approve a solution while provinces take their own view and require different sign offs. In DRC, approval processes for interventions have sometimes become stuck at province level. In Somalia/Somaliland, the different ministries of health often want their own systems, each with its own version of DHIS2 for instance. We are aware of instances where a Government has signed an MOU to take a solution to scale and then asked for a kick-back from the agency to proceed, which, when declined, meant the government went with a different solution.

Where the demand originates from government, and fills a recognised need, there is evidently greater buy-in. In Malawi, Village Reach was asked by the College of Health Sciences to develop remote training specifically for a new Pharmacy Assistants Training programme. This cadre existed on paper but not in reality. Village Reach worked with Government from the beginning to establish a suitable pre-training course which is being continued by the College.

While it is not yet fully funded by the Government, it is ostensibly owned by the government and likely to be sustained.

Example 13: MOTS - Mobile Training & Support Service, Sierra Leone Implemented by the public-private partnership EBODAC (Johnson and Johnson, World Vision, Grameen Foundation) and the Ministry of Health & Sanitation (MoHS). MoHS in Sierra Leone has developed, tested and rolled out an innovative way to provide refresher trainings to a large group of remotely located CHWs. IVR was selected as the technology to deliver audio-based refresher trainings on the topics of vaccines and outbreak response including Ebola disease surveillance procedures. MOTS was developed in close collaboration with MoHS. Training content was customized in line with the national training curriculum and case reporting requirements.

The technology was found to be readily accepted by the CHWs and their engagement was such that they also provided important elements to be improved prior to further implementation. There was however signs of general fatigue of the IVR methodology for participating in the quiz assessments which required further investigation. It was found that repetitive aspects of important training content can be reinforced without the need for additional classroom presence of the CHW community.

Sustainability requires cost containment and subsequent software accessibility for authorities.

Transparent partnership and alignment with the MoHS from the outset of this project is considered an important element of its success.

Example 14: Government-branded App ‘COVID-19 Ethiopia App’ Last Mile Health (LMH) has supported the Government of Ethiopia to produce the first MoH digital learning platform attracting many users and reaching beyond the expected numbers and areas. The App consists of 6 modules to train a broad range of frontline health workers and inform others about COVID-19. The GoE see it as a launching pad for a further culture of digital learning in health.

Because LMH had an existing relationship with MoH to develop technology-based digital learning platform for RMNCH, it could rapidly switch to developing an COVID app due to urgency. Among reported challenges, only 6% of users completed all 6 modules, possibly due to incompatible Android versions. 87% of App users are male, whereas 53% of health workers in Ethiopia are female, but mostly residing in rural areas and without access to internet. The App has been heavily promoted through Facebook which is used by more men than women in Ethiopia.

35

Mott MacDonald | Mapping and Assessing

Classified as Internal

While the need for country buy-in is self-evident, it can be difficult in practice for a combination of reasons. Ministries are large and complex organisms and support from one quarter does not signify support across the board. Information may not flow well within a ministry or there might be internal tensions or power struggles. High turnover can also hinder government engagement.

The bandwidth of busy department officials is a major issue and development partners reported difficulty in being able to access and engage government counterparts. Most development partners do take great pains to ensure government approval or sign off, which is often a formal requirement, but there is a great difference between sign off and true ownership and the solution being actively supported by a country government. Genuine engagement requires time and patience for relationship-building and to co-create workable solutions that a ministry will want to get behind.

3.7

Recognition, accreditation and lifelong learning

There is a close link between government buy-in and government health systems formally recognising and accrediting training within their health workforce development strategies. The next section discusses further the importance of expectations of those in positions of influence on providers’ behaviour. Drop-out rates of online training can be high but are much lower when training is a formal requirement of continuing certification or career development.

“BMJ Learning” is accredited in 70 countries and confirms that linking training with accreditation for continuing medical education or continuing professional development improves uptake. They report current direction of travel is to link continuing CME and accreditation. While this experience may not be directly applicable to front line workers, getting some sort of formal accreditation or recognition should help uptake.

WHO conducted an internal review and launched the vision of WHO Academy representing a complete shift from a standardised, content-driven training approach, where everyone gets the same course, to a lifelong learning strategy, enabling tailored learning to different users to acquire competency and make behavioural changes. The vision of the new WHO Academy is that regulators worldwide will formally recognise a WHO certificate. A more globally joined up approach to accreditation would certainly help portability of health workers’ qualifications.

Accreditation itself does not have an impact unless there are human resources policies that recognize and reward increased competencies. Accreditation with measurable indicators that are assessed by an independent body, similar to accreditation of academic programs, can increase stakeholder confidence in the quality of a training program. Further, if a certificate of completion of the accredited program is formally recognized as a means for career advancement and is funded as a line item in a MOH budget, quality and sustainability are more likely to be achieved. Ethiopia is a good example of a government which has worked to establish a career ladder for its CHWs (Health Extension Workers) to improve motivation and retention and in turn quality of care.

One reason that in-service training is not always linked to formal accreditation, is that it is often times driven by ministries of health or development partners who liaise with their health sector counterparts whereas the higher educational institutions that provide pre-service qualifications to the health workforce might come under the education sector, a different set of stakeholders.

Bringing the two closer together to promote a lifelong learning approach, will involve closer multi-sectoral coordination by governments as well as development partners.

36

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 15: LEAP (Amref) Kenya Leap! Is a public private partnership between Accenture, M-Pesa Foundation, Safaricom, Vodafone (Mezzanine) and the Government of Kenya which has trained over 35,000 learners across 30 counties in Kenya. This is a pilot initiative that seeks to assess the ability to enhance routine Immunization and defaulter tracing training by integrating an mHealth component that offers initial and refresher training on Immunization to Community Health Volunteers (CHVs), collaboration and supervision opportunities to improve effectiveness of the program. Leap! offers accredited healthcare training content and is customisable to needs. Operating principles include aligning fully with Ministries of Health in Africa in the empowerment of their health workforce and developing Africa’s first fully integrated mobile health platform. It is currently embarking on scaling up across various countries in Africa aiming to be the "goto" solution for health worker training in Africa and beyond.

Example 16: Health[e]Foundation HealtheFoundation provides Continuous Medical Education for HCWs covering topics such as HIV/AIDS, TB, mental health, basic occupational health, research, health problems among refugees, as well as preventive programs on sexual and reproductive health. Programs start with a kick-off workshop, followed by a self-study period, and a follow-up workshop, but can be adjusted to meet users’ needs. The e-course curriculum can be accommodated on any global open-source platform, as the e-modules are developed using the universal SCORM format. They can be accessed online via a website, offline on a USB stick and via an App. 20,000 health care workers have been trained in 34 countries. In March 2019, 110 participants from Nigeria finished HIV training - 90% passed and received certificates co-signed by the University of Amsterdam Academic Medical Centre.

3.8

Feasibility / Acceptability of Technology

The following lessons have emerged on the relevance, feasibility and acceptability of technologies commonly used in digital training or data exchange.

Offline capability is essential as most settings experience inconsistent connectivity It is often now the norm for systems to have offline functionality so that content can be downloaded in advance and not rely on consistent connectivity. Remote training works best when delivered in short bites, while bookmarking technology which enables users to return to the course where they left off is also importance for user ease.

Many apps work offline once downloaded, although the amount of storage capacity apps take up on personal devices can be a concern. Bluetooth data transfer is also being used in low connectivity environments, for example when supervisors visit remote locations and can collect data from health workers via Bluetooth.

Telehealth models cannot work offline and a good internet connection is needed for video conferencing. The success of virtual communities of practice is dependent on basic infrastructure such as electricity and internet bandwidth. However, cloud-based videoconferencing platforms are now accessible to nearly anyone with a webcam-enabled computer, tablet, or smartphone, and the growth of cellular availability in sub-Saharan Africa and lowincome countries has enabled many new applications of technology in health.

Providers prefer to access content on their own devices rather than carry separate devices

37

Mott MacDonald | Mapping and Assessing

Classified as Internal

There is a strong preference for content to be watched on providers’ own devices and this avoids escalating costs of providing hardware as programmes are scaled up. This has been reported from both low and high resource settings.

The British Journal of Medicine (BMJ) who provides online training (BMJ Learning) and decision support apps (BMJ Best Practice) has found that if people can access training on their own devices, they are more likely to use it as doctors do not want to carry two phones. Also, attention spans are short and the shorter the chunk of learning the better. 39 Where tablets are provided by programmes, aside from the cost, there can also be logistical hurdles as devices are sometimes collected at the end of each day due to high risk of theft. As ownership of smart phones increases, we would expect more reliance on personal devices.

However, there are of course many settings where video content is desired and health providers do not have smart phones. A low-cost Android tablet, connecTAB, has been used to deliver video tutorials and remote online peer-tutoring for clinical skills training. The connecTAB, which is significantly less expensive than internet-enabled smartphones, was specifically developed for areas with low bandwidth. Videos can be preloaded in order to circumvent the issue of streaming in areas with slow internet speeds, which is a common barrier.

Example 17: Low-cost Android tablets and videos to teach clinical skills in Kenya ConnecTAB, a low-cost Android tablet, was used as an educational aid for teaching clinical examination skills amongst medical students in Western Kenya. Observed Structured Clinical Examination (OSCE) scores were compared between a cohort of students who had connecTABs and a cohort of students who did not. The videos were developed by Geeky Medics, an organisation specialising in open-access clinical video tutorials. Results showed significantly higher improvements in the scores for both cardiovascular and abdominal examinations within the group who received the e-tablets as compared to the control group. The study suggested that access to connecTAB improves clinical education and efficacy and holds promise for international training in both medical and allied healthcare professional spheres in resource-limited settings.

Mobile phone training through IVR can be an effective complement to classroom training in CHWs, even in fragile states Use of voice recording is often preferred to written content particularly in low resource settings and can be well tailored to local settings. In India, BBC Media Action found only 9% of CHWs had ever sent an SMS. Recognising that they needed something simple and audio based, they created a suite of mobile health services using interactive voice response (IVR). This technology is handset independent, audio based and accessed via a simple voice call.

Viamo has developed IVR based solutions, known as its 3-2-1 service, in a growing number of countries through partnerships with mobile network operators (MNOs). These MNOs provide free airtime in return for securing loyalty to their sim card brand. IVR can work on a “pull” basis, where users dial a short code to hear a message, or on a “push” basis when health workers are called on the phones at a set time, which might be repeated as necessary for missed calls. In DRC Viamo found pick up rates rather low (30-40%), though these improved greatly when supervisors engaged and reinforced the need for training. There is a risk of general fatigue of the IVR methodology and for providers to remain engaged, hence the importance of a blended approach which involve some face to face sessions.

39

Personal communication Kieran Walsh, Clinical Director BMJ, 11 December 2020

38

Mott MacDonald | Mapping and Assessing

Classified as Internal

To use IVR optimally, recordings should be developed using iterative user-centred design processes, to consider desired speed of delivery, local dialect, rich idiomatic phrasing to ensure memorability, and emotional as well as rational arguments to make content appealing and persuasive. IVR benefits from multiple rounds of user-tests to check and finetune content.

Multi-function systems which allow functional integration are far more likely to be used Busy providers do not want to have to log in and out of different systems to perform functions.

The extent to which eLearning is integrated into other digital systems e.g. electronic health records (EHR), is of major importance and attracting growing attention. Providers are more likely to use a clinical pathway decision support app, for example, if they can link directly to this when they are with a patient and completing an electronic health record and want to check the correct tests they should be requesting. These need to be granular links which go directly to the relevant piece for clinical workflow. Integration is in fact essential for the benefits of Artificial Intelligence and Machine Learning to be realised.

It is also common sense that a wider multi-function platform that health care workers are already familiar with will be easier and more likely to be used. Familiarity and frequency of use increases providers’ confidence and skills navigating a new system which can otherwise be daunting and off-putting. Medic Mobile’s Community Health Toolkit (CHTK) provides an "end to end” community health systems platform – incorporating messaging, digital decision support, data collection, task and schedule management, patient files and analytics.

Sehat Kahani is an existing mobile app used in India which allows healthcare providers to evaluate, diagnose and treat patients via chat, audio or video. During the COVID pandemic, content has been added on to the existing app, building its potential as a wider multi-function platform that users and health care workers are familiar with. Similarly, the mobile and web application, ImTeCHO in India had a range of functions but is also used to pay performancebased payments through the App, obviously encouraging engagement and uptake of the app.

Support for change management and technological troubleshooting is vital to allow transfer of ownership Transitioning technological solutions to new owners needs time for them to bed in. Lessons from Medic Mobile, Acasus and others is that handing over responsibility for digital maintenance is best done gradually, with troubleshooting support along the way. South African digital technology social enterprise firm, Jembi, commented that digital systems are “easy to build” but the hard graft is to manage implementation and the change management needed on the ground. Jembi itself has now stopped taking on projects without a clear sustainability plan.

The NGO Village Reach has a Transition strategy with two key staff positions responsible for managing transition activities with government and relevant partners.

Living Goods, describing itself as “technology agnostic”, helps governments and implementing partners select, design, configure, and implement digital solutions that best meet local needs.

Understanding that technology alone is not sufficient to deliver health outcomes, it works to ensure ‘wrap-around’ services to build the processes and systems to ensure technology is repaired and maintained, that users adopt and own the system, and that the data is accurate and verifiable.

Example 17: Living Goods and Medic Mobile, Kenya and Uganda Living Goods and Medic Mobile have collaborated to create a set of mobile and web tools to support CHWs. The open-source Smart Health app is used by CHWs to support delivery of high quality and integrated primary health care services, including advice on COVID-19. This application also has a supervisor dashboard that enables the real-time remote management of CHWs, manages effective stocking of medical commodities, and is being used to provide the government with critical data to better plan and budget for community-level interventions.

Living Goods was involved in co-designing the solution with technology partner Medic Mobile, who has now fully handed over the technological management to Living Goods. A lesson is that handing over digital technology capability should be done gradually with partners, and build up capacity incrementally, rather than handing over everything in one go as there is always

39

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 18: Living Goods and Medic Mobile, Kenya and Uganda Living Goods and Medic Mobile have collaborated to create a set of mobile and web tools to support CHWs. The open-source Smart Health app is used by CHWs to support delivery of high quality and integrated primary health care services, including advice on COVID-19. This application also has a supervisor dashboard that enables the real-time remote management of CHWs, manages effective stocking of medical commodities, and is being used to provide the government with critical data to better plan and budget for community-level interventions. Living Goods was involved in co-designing the solution with technology partner Medic Mobile, who has now fully handed over the technological management to Living Goods. A lesson is that handing over digital technology capability should be done gradually with partners, and build up capacity incrementally, rather than handing over everything in one go as there is always troubleshooting to be done and time is needed to build up these skills.

Technology helplines have been used effectively to provide support in case of any technology related problem reported by individual users. In the ImTeCHO example in India, a lesson was that ICT support was critical for sustained support on mobile maintenance and application updates to field-based health workers. A simple, intuitive and voice navigated user interface was important for uptake, while ASHAs also needed regular supervisory feedback to ensure high and effective use of the application. Trouble shooting technology issues in the formative phase, involving ASHAs in iterative solving of App features and coaching sessions with ASHA workers for hands on support helped to ensure satisfactory uptake and adherence.

Building modular content using existing “global good” platforms supports integration, interoperability and sustainability There are various open-source platforms that have become well-established – such as OpenSRP, CommCare, Community Health Toolkit to mention just a few – and powerful new technology is continuously emerging and evolving. Many of these platforms are designed to be interoperable with other parts of the system. OpenSRP, for example, can complement and add value to other “global good” digital health information systems, including medical records systems (OpenMRS), health management information systems (DHIS2), logistics information systems (OpenLMIS), and messaging platforms (RapidPro) that are deployed at scale. Annex 5 provides a summary of the main platforms with potential for further exploitation for immunization performance tracking.

Key informants reported that it is “technically not that challenging” to make various systems interoperable, which in many cases is preferable to the alternative of dismantling functioning systems in some parts of the country in favour of a single national digital platform. Where workflows and dashboards are developed in a modular way, they can be more easily adapted to existing platforms. Ministries are more likely to need help with configuring and deploying the most appropriate solutions in the field, rather than creation of new systems.

TIP Global Health (formerly The Ihangane Project) has developed E-Heza, Rwanda’s first pointof-care digital health record which utilises real time data trends to tailor health education to individual families and to improve health care delivery. TIP is working with the Rwanda Ministry of Health to integrate E-Heza into their data-reporting systems (DHIS2 and OpenMRS) while also adding additional functions. The aim is to bring E-Heza to all health centres and CHWs in Rwanda by 2021 and all of East Africa by 2023.

40

Mott MacDonald | Mapping and Assessing

Classified as Internal Example 19: Terre des Hommes, Burkina Faso An App – leDA - co-created by Terre des Hommes (TdH) and the MoH in Burkina Faso has digitalised the WHO medical protocol Integrated Management of Childhood Illness (IMCI) and guides health personnel to diagnose sick children accurately. Data is analysed to improve the quality of care and inform decision-makers. The digital tool is built on Dimagi’s CommCare platform. Using an existing digital platform allowed Tdh to focus on the design of the digital tool rather than the platform itself, and comply with individual health data collection, transmission and storage regulations. A LSHTM evaluation found a large effect on % IMCI tasks done correctly but found no effect on improving the % of children with medicines correctly prescribed. This underscores the need for monitoring to allow continuous improvement of interventions so that they can achieve impact in saving lives.

TdH is preparing the field so that the Ministry of Health of Burkina Faso could take full control of the digital solution IeDA and directly manage it, with IeDA deployed in 67 per cent of all health centres in Burkina Faso, in two health districts in Mali and in Niger, by end of 2020.

Open source is desirable but does not mean solutions become “free” In LMICs, use of open source software (OSS) is preferred over proprietary software, for reasons of ongoing costs of licencing, adaptability and scalability. There is a misperception, however, that OSS systems are free in the longer term, when in fact the cost of the license is a very small component of the total cost of ownership of the solution and long term sustainability needs to be financed. Some countries have developed a digital health investment case to guide the development of digital health systems and motivate for funding from the national treasury for identified technologies and systems with proven healthcare benefits.

There is a need for international standards Standards for software development (interoperability, reusability, accessibility, manageability and durability) need to be considered, with implications for protecting and maximising value obtained from investment in developing e-learning. This issue has been driven by US agencies – including USAID and BMGF – and also increasingly by the UK. Since 2005, the South African digital social enterprise firm, Jembi, has been helping low resource countries develop country variants of standards, working with WHO. South Africa has been at the forefront in terms of pushing for interoperable standards and there is a new National Health Normative Standards Framework for Interoperability in eHealth in South Africa (HNSF) reflecting international best practice.

FHIR (Fast Health Interoperability Resources) HL75 standards are emerging as the most important international standard. These started to be introduced over the past 5-6 years in a slow process, building on work started with development of the Principles for Digital Development, by USAID, UN and the Rockefeller Foundation. One of FHIR’s goals is to facilitate interoperation between legacy health care systems, to make it easy to provide health care information to health care providers and individuals on a wide variety of devices from computers to tablets to cell phones, and to allow third-party application developers to provide medical applications which can be easily integrated into existing systems.

FHIR provides an alternative to document-centric approaches by directly exposing discrete data elements as services. For example, basic elements of healthcare like patients, admissions, diagnostic reports and medications can each be retrieved and manipulated via their own resource URLs.

3.9

Measurement, Individual learning and Incentivisation

The issue of measurement of training impact is widely seen as a neglected area in need of attention. It is essential for course correction, ensuring strategies can be optimally tailored to

41

Mott MacDonald | Mapping and Assessing

Classified as Internal

context, for ensuring strategies translate ultimately to high quality patient-centred care, and for ensuring cost-effectiveness of investments.

Much training is delivered by turning technical guidance into presentations, with everyone getting exactly the same content delivered over several hours or days. Learners with different needs and learning pace are thus lumped together, with the result that individual learning needs are often not met. While personalised learning is seen as priority, the challenge is delivering this at scale. Some digital education has been conceptually pedagogically weak, with more attention to the technology than learning impact. More than ever, pedagogical experts and software developers need to work closely together, to ensure that the power of digital and data analytics can be used to accurately pick up individual weaknesses or difficulties and support tailored learning. This includes greater sensitivity of measurement with feedback loops and remedial action to ensure required competencies and learning needs are met.

However current measurement practices are often limited to counting the number of training events and number of people trained, with less attention to how formal training translates into performance. Where they do exist, the most commonly available measures of impact are improved knowledge scores, measured by pre- and post-test scores. Such tests are not really proof of training impact - anyone taking the same test twice is likely to do better a second time while increases in knowledge do not necessarily translate to improved performance. While more significant than knowledge-only measures, expanded coverage is at best an indirect measure of improved health provider performance.

Directly observing competency is a more rigorous approach though requires more resources.

Objective structured clinical checklist examination (OSCE) is a type of examination used in medicine to test clinical skill performance and competence in a range of skills, and serves as a practical, real-world approach to learning and assessment.

Example 20: One Million Lives Initiative. Maternity Foundation and Laerdal Global Health The Maternity Foundation’s One Million Lives initiative - Helping Babies Survive, Helping Babies Breathe, and Helping Mothers Survive (supported by Laerdal Global Health with American Academy of Paediatrics and Jhpiego) uses a blended learning approach backed up by standardised competency assessment. During 2020 they began to turn face to face courses into an online platform which could be run anywhere in the world. As the skills being taught require hands on practice, training is a hybrid of virtual and face to face, with the trainer joining on-line but trainees assembled in groups in remote settings. The group dynamic is important as the peer-to-peer methodology involves learners practising skills together until they have mastered a skill. The OSCE style assessment has been digitised. An app shows the procedure and a checklist and learners can practice until competent and confident. Learning is measured and there is now a focus to move towards analysing data from the training to see which HCPs are learning well and which need more support.

Observing performance in a field setting is more logistically challenging. Many organisations use a structured checklist as part of supportive supervision (although interestingly evidence reviews have found mixed results on the effectiveness of such checklists). The utility of checklists likely depends on what is done with the data and on the quality of interaction between supervisor and supervisee. Ideally, data from field supervision is used to prioritise those in need of more support and identify areas for targeted support.

For non-clinical skills such as health programme design or management, it is less clear what is the practical skill to be practised and observed. During training, learners could be tasked with simulation exercises, to produce a micro-plan for a vaccination campaign, for example. However the ultimate measure of training impact will be to monitor empirically what happened following training using simple, SMART indicators.

42

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 21: PSI’s Health Network Quality Improvement System (HNQIS) The INGO Population Services International developed a Health Network Quality Improvement System (HNQIS), a scorecard to track improvements in terms of observed skills according to defined standards of care. HNQIS is an electronic tablet-based application composed of four modules that support healthcare supervisors to: (1) plan supervision visits, using a prioritization matrix that presents facility-specific quality scores and patient volume, (2) assess providers’ quality of care against clinical standards, (3) improve providers’ quality of care through tailored feedback, and (4) monitor quality improvements over time.

HNQIS works offline, so consistent connectivity is not required. Quality scores from HNQIS assessments for family planning in Cambodia, Kenya, Mali, Uganda, and Zimbabwe from 2016 to 2018 showed improvements over time. In Somalia, PSI produced digitized checklists approved by MoH. Provider-client interactions were observed and improvements recorded. 1

Incentivisation Training budgets, as commented earlier, continue to absorb a lot of available funding. Paying participants a per diem to attend training is common, based on the rationale that health workers tend to be low paid and should not be left out of pocket from attending training. However per diems have become a de facto salary top up in many settings, often subverting the motivation to attend training to something financially-driven. Per diems have become so entrenched, that where some face to face training is replaced by virtual learning with no per diems, there is resistance. Viamo reported such resistance as a cultural hurdle when rolling out remote training via IVR in DRC. In the context of low resource settings and constraints on budgets to adequately pay the health workforce, there is certainly an argument for ODA funds being channelled to frontline health workers if this can leverage quality. Not all development funds can be spent on short term “investments”, as the real challenge comes in sustaining implementation and quality delivery.

The problem is that currently paying for training or paying per diems simply incentivises attendance at training and does not differentiate according to impact on performance.

The Framework for Immunisation Training and Learning (FITL) is a shared conceptual framework developed by a range of individuals convened under the Teach to Reach initiative by the BMGF to create a better environment for training and learning. This framework stresses both measurement and feedback of both formal and on-the-job learning, to allow for continued improvement, and at the same time better coordination and alignment of funding and incentives.

This Framework emphasises Individual Learning Journeys as key to supporting the needs of managers and the health workforce. Whether referred to as user-centric learning, tailored learning or individual learning journeys, these concepts are gaining a lot of traction.

Strengthening individual learning involves fostering, recognising, and rewarding self-directed learning in the workplace, as well as offering post-training, on-the-job support to encourage and reward retention and programme improvement.

Other than directly observing performance, either during training or in the field by a supervisor, another option is to obtain feedback from users of a health system. Given the importance of health systems delivering high quality services, which defines quality to include the user experience, capturing these perspectives could be an important part of measuring performance, along with other types of data. This might be especially useful for measuring provider attitude or behaviour given the inherent bias when he or she is observed. Digital solutions that exist for training also offer potential for soliciting community feedback. For example, Viamo’s 3-2-1 platform is already used for survey data collection. If needed, respondents can be incentivised

43

Mott MacDonald | Mapping and Assessing

Classified as Internal

to take part with a small amount of phone credit, and this type of data collection is very costeffective and scalable. The anonymity of a mobile phone response can also solicit more honest feedback with less risk of courtesy bias. There are many other firms – GeoPoll, 60 Decibels and more – specialising in rapid cost-effective real time data collection to support agile and adaptive programming.

Example 22: MomConnect for user feedback, Praekelt MomConnect is a multi-faceted programme that creates demand for maternal health services as well as improves the supply and quality of those services. It includes stage-based health messages developed by the National Department of Health with support from its partners at Baby Center, a text-based helpdesk that provides answers to pressing questions, a library of health information accessed via a USSD menu, and a service rating feature that allows subscribers to report on the quality of services they receive at facilities. Integrating directly with the national health system – with partners at Jembi and HISP – MomConnect presents decision-makers with control interfaces and dashboards to ensure that feedback from mothers reaches the highest levels of government.

With diverse methods available to measure training impact and provider performance – from structured clinical assessment, field observation, community feedback, as well as HMIS data – there is scope to replace some of the current training expenses - costly hotel, catering and per diems - with alternative funding channelled to frontline health providers based on transparent, data-driven performance measures. Indeed, where training pivots from being mostly classroombased to greater reliance on digital channels, there are savings which could be distributed evenly. A win/win scenario would be for health providers to enjoy benefits in terms of more efficient use of their time and less time spent away from their homes, with an opportunity to earn “top-up” payment based on actual performance.

44

Mott MacDonald | Mapping and Assessing

Classified as Internal

4 Future Trends and Emerging Technology and Innovation According to WHO40, future trends in digital education include, but are not limited to: • • • • • • • •

4.1

virtual, augmented, mixed reality in education for skills development, clinical scenario role playing and facilitation of procedures; personalized learning adapted to the health needs of individuals and populations; AI and intelligent tutoring systems facilitating the customization of learning experiences; SGG for simulation and incentivized learning; utilizing social media or communication platforms, such as Facebook, WhatsApp, Skype, LinkedIn or Twitter, for knowledge sharing and interdisciplinary communication; big data and learning analytics (including regulatory and security issues) to determine areas needing reinforcement MOOCs, virtual learning environments (VLEs) and learning management systems as part of lifelong learning (continued medical development/CPD programmes; developing standards for reporting digital health education intervention trials; and validity evidence on measurements of instruments used in digital health education trials

Virtual reality and augmented reality

Given the finding that training which takes place in a health worker’s workplace setting tends to be more effective, there is a challenge for digital training to capture this “real world” dimension in so far as it can. Some of the newer trends in the training and development of healthcare workers include virtual and augmented reality, which are immersive and interactive technologies that allow the user to explore a digitally generated 3D world in real time. They provide real life simulations and allows for hands on, active learning and skills development experience to be used in clinical practice, far superior to traditional and theoretical learning.

A systematic review and meta-analysis41 on the use of virtual reality for the education of health professionals published by the Digital Health Education Collaboration in 2019, found virtual reality to improve knowledge and skills outcomes of health professionals when compared with traditional education or other types of digital education such as online or offline digital education. The Digital Health Education Collaboration have also published results from a systematic review and meta-analysis that suggested virtual patient simulations can more effectively improve health care workers’ skills development when compared to traditional learning.

40 41

Digital education for building health workforce capacity. Geneva: World Health Organization; 2020. Licence: CC BY-NC-SA 3.0 IGO. Kyaw BM, Posadzki P, Paddock S, Car J, Campbell J, Tudor Car L. Effectiveness of Digital Education on Communication Skills Among Medical Students: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019 Aug 27;21(8):e12967. doi: 10.2196/12967. PMID: 31456579; PMCID: PMC6764329.

45

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 23: Life Saving Instruction For Emergencies Using virtual reality for medical simulation training, a team of researchers based at the University of Oxford and KEMRI - Wellcome Trust Research Programme (KWTRP) - Kenya, developed the Lifesaving Instruction for Emergencies (LIFE) project. This project was tested on healthcare workers in Kenya to establish the potential feasibility and acceptability of low-cost VR for medical simulation training in a low income setting. The technology was well adopted by the cohort however the high cost element due to the hardware needs remained a barrier to wider adoption and scale up.

Trainees work through current best practice guidelines in a ‘real’ environment - they identify and find equipment, demonstrate knowledge of how to assess a newborn baby, and go through the key steps of basic resuscitation in the correct sequence, all under a realistic time pressure. The LIFE app now includes training scenarios that help healthcare workers manage and treat cases of COVID-19. The LIFE app is available on the Google play app store, has received a 4.5 star rating from users and the following review - 'It feels like a real life situation and helps you think on how best to save life in the shortest time'. The LIFE project has been supported by many global health players including the Médecins Sans Frontières, USAID, DFID and the BMGF. Interestingly, a University College London start up called ‘Musemio’ are using the virtual reality platform combining virtual reality with gaming to develop innovative mobile virtual reality games for educational purposes for children and aids in the growing remote learning needs that have arisen out of the COVID-19 pandemic. The inventors have found that the immersive experience improves retention of knowledge, helps with attention spans and can help children to understand and relate to a subject better. Although they are active in the education space and not the health sector, the start-up has received a number of small funds and has been named ‘ones to watch in world of virtual reality’.

The WHO Academy is a flagship initiative still in development. The Academy aspires to be a globally accessible school for the future, combining the latest technologies in digital and remote learning (AI, VR etc) with advancements in adult learning science to offer innovative, personalised and multilingual training. The main goal is to ensure training is provided in a more systematic way to achieve scale and impact and reduce the typical 10-year timeframe to implement guidelines. Through a hub and spoke model featuring digital plus on-site training in campuses around the world, the Academy will utilize a train the trainers and leaders/managers approach allowing it to reach other benefactors with multiplier effect.

The WHO Academy launched its first augmented reality course for healthcare workers, on the proper use of personal protective equipment to protect themselves and patients during the COVID-19 pandemic. The course is intended as an immersive, engaging and easily accessible learning experience, which can be downloaded via a smartphone from the Apple App Store or Google Play Store, globally and can be completed in 20 minutes. The WHO Academy hub in Lyon, France will feature a health emergencies simulation centre and collaboration spaces for learning co-design, research and innovation.

4.2

Serious Gaming and Gamification

Serious gaming and gamification (SGG) refers to approaches in which learners engage in a competitive activity with educational goals intended to promote knowledge or skill acquisition.

Gamification can be defined as the application of the characteristics and benefits of games to real world processes or problems and serious games can be defined as games designed specifically for the purpose of providing health professional education via a digital device. 42

42

Digital education for building health workforce capacity. Geneva: World Health Organization; 2020. Licence: CC BYNC-SA 3.0 IGO.

46

Mott MacDonald | Mapping and Assessing

47

Classified as Internal

A systematic review43 was done to evaluate the effectiveness of serious gaming and gamification interventions for delivering pre- and post-registration health professions education which covered 30 studies. From this review, although only one study measured patient outcomes (and reported better scores for doctors controlling blood pressure in some subgroups), all the individually played games with an objective assessment of knowledge suggested serious gaming/gamification was superior to traditional learning.

The review concluded that SGG is an educational strategy that could contribute to transformation of health workforce education in both pre- and in-service training, although there was lack of evidence from low- and middle-income countries (LMICs) which limited the review’s applicability. There was also a lack of studies which assessed patient outcomes or provider behaviour.

The cost of serious gaming devices might be a barrier for use compared with other digital approaches. For example, some use game consoles, which many health care workers in LMICs would not have access to. Other approaches include lower-cost modes of delivery, such as projecting a serious game to a group of students who play together. However, none of the eligible studies provided any information about economic outcomes of education or adverse or unintended effects of the intervention, which limits our understanding of the feasibility of implementing these interventions in practice as a cost-effective solution.

The OpenWHO platform, launched in June 2017, equips frontline responders with knowledge to better contain disease outbreaks and emergencies. It is currently exploring possibilities to transform simulation exercises into interactive games, digitizing live role-playing game exercises online for epidemics and health emergency work. The PPE app has a new feature that deploys augmented reality (AR) technology.

Coach2PEV – is an innovative digital solution rolled out in Senegal and Benin by GaneshAID to measure EPI performance. Part of the approach uses gamification, performance reward and gamification, where health workers can see their performance ranking in a mobile phone dashboard.

Technology-focused providers such as Viamo use socalled “Edutainment” to reach health providers and the public. In 2020 Viamo rolled out its Wanji Game about COVID-19 prevention in DRC, in partnership with Peripheral Vision International (PVI) and JHUCCP Breakthrough ACTION. This interactive “listen-thenchoose” audio game is designed to promote positive behaviour and to test players’ knowledge and understanding of prevention techniques.

In high income countries, the use of games is commonplace in management and leadership training as the skill being taught - such as empathy, listening can be better taught through experience than imparting theoretical content.

43

Serious Gaming and Gamification Education in Health Professions: Systematic Review. Sarah Victoria Gentry1 et al.

J Med Internet Res 2019 | vol. 21 | iss. 3 | e12994

Mott MacDonald | Mapping and Assessing

Classified as Internal

Example 24: Project Nigel An example of a non-digital learning game used by a UK professional performance and coaching company, Lane 4, is a game called “Project Nigel” – in which three tiers of staff (workers, managers and senior executives) are required to complete a task collectively, while being given three different briefs and sets of instructions and working in relative isolation from each other. The point is to bring home the frustrating impact of lack of communication, lack of consultation and failure by leaders and managers to use the inherent knowledge of the “workers” to complete the task. Given many of the issues are management/leadership issues which are common to many sectors, there is scope for adaptation, and possibly digitization, of such illuminating games to local contexts.

The education platform, Moodle, is an open source Learning Management System (LMS) used by universities in the north and the south. Among its varied and versatile functions, the platform allows organization of outdoor/indoor serious-games with learners as its “TreasureHunt” module supports virtual-map treasure-chases using geolocation and QR codes. Medic Mobile’s Community Health Toolkit platform uses Moodle as its back end with offline use and gaming features, integrated with caregiving applications.

4.3 4.3.1

Remote learning and COVID adaptations in other sectors Education Sector and COVID Adaptations

In the education sector, remote learning has a long and extensive history. The adoption of new technologies has brought many benefits, including automated response software; making accessible authoritative and up-to-date information and learning resources in multiple media; enriched self-access interactive learning activities; increased peer-to-peer interaction and communities of practice; real-time tutorial support; new learning assessment opportunities; and administrative communications and record updating.

However, the benefits associated with these developments are constrained by limitations in internet connectivity, the familiarity of providers and consumers of teaching and training with technology, the cost of data transfer, electricity supply, ownership of or access to suitable hardware and software. Thus for remote learning, older distance education approaches continue to find a use to reach learners, including the use of television and radio broadcasts.

Where internet access allows, there are also options of accessing video or audio files asynchronously, as podcasts or videos to download or stream.

Similarly, audio or video can be loaded onto memory media, such as SD cards, USB flash drives, and provided to learners to install on their phone, tablet or laptop, as relevant. This avoids dependence on internet connectivity and data download costs. Thus the longstanding and evolved model of Interactive Radio Instruction (IRI), emerging from a mathematics teaching programme in Nicaragua in the 1980s and supported in multiple countries, has spawned Interactive Audio Instruction (IAI). An example of a model with installed video clips and other materials is English in Action, in Bangladesh. This programme provided demonstrations of teaching techniques and English language practice for teachers. Its predecessor, of the same name, in South Africa, used radio broadcasts transmitted directly into classrooms, for teachers to use with their pupils, live, in conjunction with pupils’ textbooks.

Whatever communications media and combinations are used, widespread preference is that such remote access to learning resources should be supported by face-to-face or if that is not possible, online chat discussion groups, whether self-supported or led by facilitators. Overall, rather than purely remote learning, the most effective option if possible is widely felt to be a blended approach with some group or individual interaction.

48

Mott MacDonald | Mapping and Assessing

Classified as Internal

Remote learning under COVID-19 Against this pre-existing background, the COVID-19 pandemic has provoked an accelerated and wholesale ‘pivoting’ (i.e. radical and rapid shift) to remote learning, worldwide, at all levels of education systems. Organisations, teachers, trainers and learners with little or no background experience in remote learning have been forced, or have chosen, to adopt technology-based methods as their existing methods of operating were abruptly made unavailable.

As well as the direct teaching process, this has equally impacted on the essential supporting functions of education programme management, course development and production, administration, assessment, monitoring and certification. It has required new staff training, in areas including materials design and development, online tutoring, counselling, pedagogy, record keeping. Staff and learners have needed to strengthen and learn new ICT skills.

Uses and technologies have proliferated. International bodies have mobilised to collate, share and advocate experience and resources in remote learning. Notable among these have been the World Bank, OECD, EdTech Hub, the INEE, UNESCO and FCDO through Frontier Technology. These have provided platforms for interested parties to identify and contact organisations and programmes with potentially useful and relevant information, experience, expertise or resources.

Different models of remote learning have been adopted in different locations as an emergency response to the closure of schools and colleges arising from the pandemic. To illustrate this range, typical examples are below.

In Ghana, pre-service teacher training shifted from face-to-face model to a distance learning model when the universities and teacher training colleges closed. With the assistance of a project, T-TEL, which was supporting the Ministry of Education to enhance its teacher training, all teacher training went online. Teacher trainers were trained in online teaching and learning, students continued studying, including undertaking and submitting assignments. Training materials were accessed from a micro-site online. Participating teacher training colleges formed WhatsApp and Telegram groups for their students. Assessment and tutor-student interaction continued, entirely using remote methods. The training materials were loaded onto SD cards and provided to students who had limited internet access. For students who did not possess a smart phone to enable them to study this way, suitable handsets were made available on a hired-purchase basis.

In Uganda, all schools closed. With the support of an ongoing education development technical assistance project, SESIL, radio programmes were prepared centrally by the National Curriculum Development Centre, to reflect the school curriculum, and then translated into regional languages and broadcast by regional radio stations, aimed at the school-age children who were out of school. As well as the direct broadcasts to pupils, there were parent radio programmes, to encourage and advise parents and carers to support their children in learning.

Based on initial evaluation feedback, radio programmes were revised to include scenarios of a teacher with pupils in a class; and following each broadcast episode, there was a live call-in for questions and clarifications.

In Maldives, when schools closed, the Ministry of Education carried out an emergency review and streamlining of the school curriculum, so that it was reduced in scope to give a better possibility of attending to key essential parts by the reduced means of remote learning. Two approaches for lesson delivery were adopted in parallel.

•

Firstly, television programmes were made, of teachers delivering the lessons in a classroom (later, in their home, in lock-down). The recorded classes were then broadcast to a published timetable, live on TV, and also uploaded to a website for

49

Mott MacDonald | Mapping and Assessing

Classified as Internal

•

download or streaming on demand. A similar approach was adopted in several other countries.

Secondly, Google Suite was adopted and teachers switched to holding live classes with their usual class of students, to a fixed timetable, by Google Classroom. Homework was set and received back, and marks recorded. Additionally, through the same system, attendance records were kept, teacher timetabling and absence and substitutions arranged. Teachers sought out and used teaching resources – materials and Apps, such as Pear Tree interactive presentation software, at their own discretion. Teachers were trained and certified as Google trainers, to support teachers in their locality. This approach was made feasible due to students from P3 upwards having been provided access to a tablet computer.

While there has been this wholesale shift to digital delivery, some analysts have described this as more digital “conversion” than digital “transformation” particularly in higher education establishments. In the dawn of the digital explosion last year, many were caught off guard and reverted to simpler tools, for example delivering lectures through live, on-line sessions. However digital delivery also brings its own pedagogy and new ways to do things, extending boundaries of time for pre-class and post-class. For example, a “flipped classroom” is where students can listen to a pre-recorded lecture first on their own, then might be asked to take a test to gain entry to a live classroom, to demonstrate sufficient understanding of the subject matter. The class can then focus on the more interesting, discussion and analysis, after students have absorbed the factual part. This is a very good way of maximising the potential that digital has to offer.

Though it is too early to see how matters will change following the present COVID-19 situation, there is a widely expressed view that there has been a permanent change and that ‘building back better’ in the education system will include the incorporation to some extent of remote learning, alongside restoration and strengthening of face-to-face learning. For some training purposes, which are not logically tied to a locality in the way that school-based learning is, there may be less requirement to return to face-to-face mode. However, this would only be so if remote means were found to be sufficiently effective.

4.3.2

Agriculture sector

COVID-19 is impacting most sectors due to the mitigations, such as social distancing and lockdown measures, implemented to reduce further spread of the virus. The food and agricultural sector and its workers have been exposed to the impacts of the pandemic and like many other sectors have applied innovative and digital technology to respond to challenges.

The agriculture sector has several online information sharing platforms, predating COVID-19, to enhance the knowledge base of agriculture workers. In addition to online learning platforms, agricultural workers have also benefited from innovative technology that helps improve worker performance and efficiency, especially during COVID-19. An example is the Water Efficiency Pilot based on Smart Agro technology that are being implemented in some Latin American countries. The project consists of a combination of remote sensors in the crops to detect agronomic variables, matched with an artificial intelligence algorithm to predict optimisation factors. The workers then receive recommendations via a smart device and can plan when to be in the field to irrigate their crops.

Taroworks44 is another example of a tool used to remotely manage agricultural field work operations. Through an offline platform and mobile app, Taroworks enables the creation of feedback loop between the extension office and field agents on farm data, crop yields and training materials. During the pandemic, the mobile app has been used to conduct field worker training and keep remote team members updated on important developments, via videos,

44

TaroWorks Mobile CRM Streamlines Field Operations

50

Mott MacDonald | Mapping and Assessing

Classified as Internal

documents or spreadsheets. Similarly, the Olam Farmer Information System 45 works as a monitoring and evaluation tool for small agricultural companies, and the newer apps built in the system provide agri-training and development for workers, which has been used by over 300,000 farmers across 60 countries.

The technology solution providers Viamo have developed IVR messages for both teachers and agricultural extension workers in DRC, in addition to their work in the health sector. Viamo has produced IVR messages through its 3-2-1 service. Farmers can listen to sequenced recordings on topics such as improved seed varieties, how to purchase, how to plant, cultivate, and can hear impact stories. There are also early warning messages related to shocks, pests, weather etc., messages on climate mitigation and on market prices.

In Nepal, in response to an issue caused by high yields and insufficient storage capacity, Viamo launched an app to connect agricultural buyers and sellers. Users could register, select whether they had access to transport or not, and be connected to the best available option that matched their needs.

Viamo’s 3-2-1 service is sustainable as it is funded through partnership with mobile network operators (MNOs) – this PPP is not donor dependent. Viamo has found that putting a small price on messages can increase their perceived value. There can be different ways to do this – in Malawi, messages were free but users had to have a certain balance of credit on their phone to access them.

45

Olam Farmer Information System (olamgroup.com)

51

Mott MacDonald | Mapping and Assessing

Classified as Internal

52

Mott MacDonald | Mapping and Assessing

53

Classified as Internal

5 Country Case Studies Looking more closely at a selection of countries, the following section provides a snapshot of specific contexts and experiences to add to the wider mapping. Country case studies are informed by key informant interviews at country level and web-based desk research. Digital developments move fast and with that so do respective behaviour changes in the population with their use of the technology. Even in rural settings lacking infrastructure, solutions which were not appropriate 5 years ago are now becoming more possible. Such is the proliferation of initiatives in the digital health space, these case studies are not intended to be comprehensive country scans, but rather some indicative findings focused on interventions Gavi are likely to be interested in. We look at issues relating to aid effectiveness – country ownership, harmonisation and alignment (and interoperability of systems), managing for results (data quality and analysis) and mutual accountability.

Country selection Countries were selected in agreement with Gavi based on several criteria: digital health maturity, high zero dose children, and covering the spectrum from fragile to transitioning countries; geographic spread.

Countries become eligible for Gavi support if their average Gross National Income (GNI) per capita has been less than or equal to US$ 1,630 over the previous three years (according to World Bank data published in July each year). When a country crosses the eligibility threshold, it enters the accelerated transition phase and starts to phase out of financial support. Indonesia was our ‘transitioning’ country.

Country

Gavi total disbursements (2000-2019), $

DTP3 coverage % (2018) (official country estimates)

Pakistan

1,316,193,881

72

Ethiopia

1,065,480,383

95

Somalia

45,415,377

Unavailable

Rwanda

159,879,380

97

Senegal

152,774,993

81

Democratic Republic of the Congo (DRC)

894,611,352

94

Indonesia

193,738,779

76

Scoring on the Global Digital Health Index The Global Digital Health Index is an interactive digital resource that tracks the institutionalization of digital technology for health across countries. Still in its early days, it includes a selection of countries that opt in to participating and being assessed.

Country scorecards produced in 2019 assess digital health against a range of areas, scoring them from 1 to 5 in a number of domains including health workforce training in digital health.

There are some interesting results. For instance, Ethiopia with its high levels of government ownership and focus on digital health in pre-service training, scores very well in the workforce domain, whereas Pakistan, despite having high levels of digital infrastructure and in-country

Mott MacDonald | Mapping and Assessing

54

Classified as Internal

expertise, scores relatively poorly as it has not institutionalized training in digital health within pre-service or in-service health workforce training.

Institutionalization of Digital Technology for Health Workforce Country

Workforce Digital Health Phase (1 lowest, 5 highest)

Overall Digital Health Phase (1 lowest, 5 highest)

Pakistan

1

2

Indonesia

2

3

Ethiopia

4

3

DRC

2

2

5.1

Pakistan

Overview Digital interventions for vaccinators and community health workers have proliferated in recent years in the areas of immunisation, maternal and childcare and diseases tracking. Two notable immunisation performance initiatives have been scaled up in partnership with the Expanded Program of Immunisation (EPI). Pakistan is a decentralised country with the Federal EPI Program under the Ministry of Health providing a coordination role, with provinces mandated to adopt, implement and fund digital technology. Most digital health activities in Pakistan have been initiated at the provincial level and presently efforts are underway to consolidate into a single central digital MIS for the EPI. A national digital health strategy is also under development that will support the National Health Vision 2016 - 2025. National level dashboards, registries, and other strategies have been initiated, and will be expanded in the coming years. 75% of the population report to have some form of mobile phone46 and 15.9% reported to have a smartphone in 201947 (with the figure rising all the time).

Example Interventions Intervention

Description

EVACCS

Implementer: Acasus, Punjab Health Department, Punjab Information Technology Board (PITB) and the World Bank Detail: Smartphone-based vaccinator monitoring application. The PITB gives all community-based vaccinators a low-cost mobile phone with the EVACCS app installed enabling them to report their immunisation activities by recording community visits using global positioning system (GPS) co-ordinate check-ins and photo uploads. Data uploads are managed by the Punjab Information Technology Board.

Reports are displayed on dashboards using polygon-based maps that turn green when a vaccinator conducts immunisation activities in a community, driving both vaccinator performance and political accountability.

Scale: Dramatic improvements in immunisation coverage (up to 96% in Punjab) – primarily due to improved vaccinator performance and area coverage. In 2018, 75% of all 12-13m children in Punjab were fully vaccinated, up from 61% in 2014.

46 www.datareportal.com, January 2020 47 Newzoo's 2019 Global Mobile Market Report, 2019

Mott MacDonald | Mapping and Assessing

Classified as Internal

Zindagi Mehfooz

Implementer: Interactive Research and Development (IRD) Detail: IRD and The Indus Hospital (TIH) provide technical support for the implementation of an Android phone-based immunisation registry to record and analyse program data in real time Scale: All 29 districts in Sindh. Registered over 2.2 million children under 2 years of age and 0.7 million pregnant women over 2017-19. Used by more than 2,200 government vaccinators.

Immunisation Roadmap

(Governance)

Health worker pilots for immunisation

Implementer: Acasus, Gavi (funder BMGF) - 2019 onwards Mott MacDonald, Acasus (funder DFID) as part of TRF+ - 2014 – 2018 Detail: Immunisation Roadmap is a supervisory application for use at tehsil and district level, which then feeds into a dashboard with digital data packs and portals provided to district and provincial governments. Focused only in Polio high risk union councils. A larger PHC Roadmap initiative for Punjab and KP provinces provides digital data on immunisation and maternity care for provincial planning: integrates independent assessments of health facilities, EVACCs data.

Implementer: Aga Khan university, Gavi – research trial: 2014-18 Detail: Android Immunisation Registry supported by vaccinator/ LHW training, peer support, district recognition and performance-based fuel transport awards Scale 3 districts

Sehat Kahani Mobile App

Implementer: Implementing partners: Ministry of Health; Zetsol Technologies; 10 Pearls Pakistan; Adamjee Insurance.

Detail: Allows healthcare providers to evaluate, diagnose and treat patients using telemedicine app via chat, audio or video. Patients can create their personal E-Health Record comprising of previous medical history, which is also visible to the connecting doctors. Adapted to COVID-19: suspected cases are forwarded to the relevant authorities helping to track the number of cases in-country.

Scale Over 50,000+ downloads, and over 18000+ consultations for both general medical complaints as well as COVID-19 concerns. Of these, 300+ individuals have been referred onwards as potential COVID-19 cases.

The app has strong government buy-in and has been invested in by WB, USAID and UKAID. The COVID work has been added on to the existing app so it has the potential as a wider multi-function platform that users and health care workers are familiar with.

Technological uptake Vaccinators and lady health workers in Pakistan have had little issue in navigating mobile applications. Most frontline health workers were already either using android applications and social media platforms, or simpler mobile phones. However, there are issues with data manipulation by vaccinators with both immunisation tracking applications. Vaccinators were sufficiently technologically savvy to re-work the immunisation GIS tracker to show visits had taken place, and these were picked up only in separate field validations exercises by EPI programs. Digital use capacity is weak amongst government managers, particularly mid-level managers in district health offices and provincial EPI programs. So far there has been little investment in terms of dedicated IT posts and digital systems for EPI programmes. At the senior

55

Mott MacDonald | Mapping and Assessing

Classified as Internal

executive leadership level there is considerable policy appetite for digital data and dashboards for oversight and operations planning.

Smaller scale pilots that co-produced the application and invested in district performance accountability and recognition of vaccinator tracking had positive feedback from health workers.

Vaccinators and LHWs report that the use of mobile apps improves self-imaging in the community, helped identify defaulting children for timely follow-up visits, peer recognition and support, as well as communication with other vaccinators/LHWs for experience sharing and trouble-shooting. There is good compliance from health workers without any financial incentives. Recognition by supervisor and the community was experienced to be a powerful incentive, and the fear of being tracked and that somebody may be looking at the data also spurred performance.

Data Attributes Immunisation performance and compliance with visits is calibrated differently by the two vaccinator applications in place – EVACCS and Zindagi Mehfooz - making it very difficult to get a coherent national picture. There are also concerns that ZM’s vaccination data does not tally with field verified reports and this has led to hesitancy in scaling further.

The two applications are not interoperable and national stakeholders feel that consistency is required in terms of what is measured and in reporting methods. Vaccinator manipulation is more widely seen in districts where immunisation accountability is weak, especially in Sindh and Baluchistan. There is wider ownership and trust for the simpler EVACCS application produced by the government than for the more sophisticated, externally funded ZM application which was developed by the partner without government inputs. This is a good example of the need for codesign and co-production. Field monitors can play a critical role in providing accurate, quality data, improving delivery of immunisation micro-plans and need to be included in further rollout of digital technology for immunisation. Linking of digital applications to the household and births registration national database (NADRA) is recommended for improving data validity for immunisation planning.

Immunisation Governance Use of data from ongoing mobile applications has had a variable experience in Pakistan when tied into larger immunisation governance experiences. While digital data is generated by EPI programmes and implementation partners, ownership by administrative leadership is necessary for performance management and use. Immunisation digital roadmaps help generate some level of healthy competition amongst districts to improve immunisation performance, but ongoing strong leadership support is instrumental in sustaining gains. Leadership change within the government hierarchy and within downstream district health systems affects use of data.

For example, leadership change in Punjab has negatively impacted the use of EVACCs for immunisation performance accountability, whereas in Baluchistan, fragmented provincial leadership results in poor data compliance by districts. Frequent transfers of district health officials means staff do not have sufficient time to get familiar with digital interventions and this compromises ownership. Taking vaccinators to task on immunisation coverage has faced challenges and backlash, due to political patronage enjoyed by the health workforce in several districts.

Sustainability While there is widescale acceptance of digital interventions, the absence of dedicated government funding and in-house digital expertise compromises sustainability. Government typically budgets for extra vaccinators but not for implementing digital innovations, as development partners usually foot the bill. There is some progress as immunisation activities have been recently transitioned to the long-term recurrent budget, but budgeting for comprehensive digital innovations also needs to be reflected. A consolidation of applications

56

Mott MacDonald | Mapping and Assessing

Classified as Internal

rather than a proliferation of parallel ones that don’t ‘speak the same language’ is also required to have unified health workforce training and reporting. NGOs are cautious regarding Gavi funded scale up of digital innovations preferring home-grown applications overseen by the government. Government stakeholders, while in favour of digital technology for immunisation, feel that they do not sufficiently challenge donors with alternative strategies/ solutions.

Interoperable platforms such as UNICEF’s RapidPro (which can operate with DHIS2 as well as social media messaging platforms such as Facebook Messenger and WhatsApp) have shown potential and offer a sustainable approach – for instance, in 2018 RapidPro was used in reaching 37 million children in 163 districts through real-time information to increase demand for routine immunisation with an initial focus in poorly performing polio tier 1 districts.

5.2

Ethiopia

Overview The Ethiopian government has taken a strong lead in advancing digital health. It is one of the four pillars of their Health Sector Transformation Plan and is referred to as the ‘Information Revolution’. Central to this has been a radical shift from traditional methods of data utilisation to a systematic information management approach powered by a corresponding level of technology.

As seen from the Global Digital Health Index table, the government has focused on the enabling environment of digital health and has prioritised training of the health workforce, right down to community level. As some of the interventions below highlight, the government has also taken on strong ownership and involvement in vertical programmes with their sustainability and interoperability in mind. They have very much pushed digital advancements and insisted that donor plans are aligned to their overall vision and helping to ensure the interoperability of systems and sustainability of initiatives.

In August 2020, the MoH with support from the Gates Foundation and JSI’s Data Use Partnership, launched the Digital Health Innovation and Learning Centre (DHILC); a place where health professionals can design and validate digital health tools, synthesize and promote best practices, and scale-up innovations.48 JSI are very active in the digital health space, leading the USAID funded Digital Health Activity (DHA), a five-year project that supports Government deliver their Information Revolution Roadmap. JSI work with Dimagi, IntraHealth, Orbit Health, Simprints and Websprix to deliver this work.

41% of the population report to have some form of mobile phone49 and 11.2% reported to have a smartphone in 201850.

Example Interventions Intervention

Description Implementer: Last Mile Health

48 Ethiopia Launches Digital Health Innovation and Learning Center - JSI 49 www.datareportal.com, January 2020 50 Newzoo's 2019 Global Mobile Market Report, 2019

57

Mott MacDonald | Mapping and Assessing

Classified as Internal

Community Health Academy

Detail: Supporting the MoH to improve health worker training by deploying mobile phones equipped with high-quality digital content to frontline and community health workers. The Academy is developing a global blended learning curriculum for CHWs that can be easily adapted to local contexts. The curriculum will be offered as a public good for Ministries of Health worldwide to adapt and localize, and will be aligned to WHO guidelines, protocols and global standards for quality.

Scale: Started with health workers in Addis, now expanding to CHWs outside Addis and outside health centres. Recently requested to re-start HEW training and expand to 100 districts.

COVID-19 Digital Classroom

Implementer: Last Mile Health and partners including Medical Aid Films Detail: Brings together global leaders in community health systems strengthening, training, content development and communication to provide high-quality digital health education content for community-based health workers. Uses national guidelines and has free airtime for health workers. Available offline too.

Scale: Also, in Liberia, Sierra Leone, Uganda & Malawi.

Supportive Supervision

Implementer: Acasus Detail: Health facility independent monitors are given phones and trained so that they can monitor facilities on immunization data. This data is fed into a dashboard that is managed by Acasus and that is reported to Regional Health Bureaus as part of performance management.

Scale: Potential for scaling as indicators and locations can be quickly added, although hardware costs will add proportional cost.

Digital Health Activity

Implementer: JSI with Dimagi, IntraHealth, Orbit Health, Simprints and Websprix to deliver this work.

Detail: USAID funded five-year project that supports Government deliver their Information Revolution Roadmap.

eCHIS

Implementer: Dimagi/CommCare with Federal Ministry of Health Detail: Electronic Community Health Information System. Using CommCare, the previous manual system was digitised. The app serves as a job aid for HEWs allowing them to review household and individual information, to respond more effectively. The app improves data quality and analytics and facilitates referrals.

Currently includes RMNCH modules with malaria and TB to be added, and there are urban and pastoralist versions.

Scale: Implemented in 1,250 rural health posts. There are plans to scale up in all agrarian, pastoralist and urban health posts. Training planned for Master TOT, TOTs and end users.

Technological uptake The Community Health Academy (CHA) work has been promoted through videos and social media influencers and as a result has attracted much more use than anticipated (more than 10k downloads of the app). Use of the app has extended beyond health workers, and it is seen as a potential launching pad for a culture change to widespread use of digital learning for health.

There have been issues though: only 32% of users are female - most female rural health workers are unable to access the internet; only 6% completed all modules suggesting problems with compatibility and/or engagement; and nearly one third had issues with internet access when using. Learning from this, CHA has tweaked content (using more animations), marketing it

58

Mott MacDonald | Mapping and Assessing

Classified as Internal

more to women, focusing more on HEWs with RMNCH modules, as well as researching why learners drop off.

In 2016, a Johns Hopkins’ review of mobile technology reported on feedback from HEWs that power/electricity access was the main barrier in mHealth interventions, especially for those using smartphones, with shorter battery life.51 Governance Acasus’s work enjoys good levels of government engagement, seeking to strengthen governance and accountability whilst building technological capacity. Acasus is responsible for training supervisors who are provided with tablets to monitor facilities. Incoming data is fed into a dashboard that is reported to Regional Health Bureaus to strengthen data quality, coverage and accountability. Engagement and leadership of government ensures good levels of accountability with government driving the need for interventions, ensuring interoperability between systems and using data/results which emerge.

Sustainability The Ethiopian government has a high level of interest in and is taking ownership of the mainstreaming of digital health enabling good prospects for genuine sustainability. It will still require coordinated funding decisions to ensure that established effective interventions continue to receive the necessary support – with the Ethiopian government ideally taking on more funding responsibility over time.

Across the interventions reviewed, government has, critically, been involved right from the start.

As a result of that, implementers also appear to be well connected, both directly with government but also between themselves. JSI, with the largest digital programme, appear to be the common thread that helps connect them together. LMH are part of this programme, helping to ensure compliance and interoperability between different interventions.

The work Acasus is doing sits somewhat outside this main circle of stakeholders. Capacity building is central to their work and they have a focus on sustainability, but it is too early in the process to assess how sustainable it will be in the longer term. There will need to be a gradual transfer of capacity to avoid the work needing to be taken on by the same/different donors at the end of each funding cycle.

A COVID-19 app developed by Last Mile Health/Community Health Academy had government involvement, linked to an existing partnership providing digital health modules on RMNCH. Much like the eCHIS app that uses the CommCare platform, this app is government-branded. It was launched using government-generated content based on the national guidelines and free air-time incentives. This is facilitated in the case of Ethiopia as government owns the telecoms.

5.3

Somalia

Overview Somalia is a Federal country with several regions pushing for some measure of independence.

Somaliland has already gained this in all but international recognition. Consequentially, the relationships between the regions are often competitive and can directly prevent cooperation in areas where it is needed, such as in sharing health data.

Nevertheless, Somalia offers some of the most technologically advanced and competitively priced telecommunications and internet services in the world. The numerous telecom 51 Mobile Technology in Support of Frontline Health Workers - A comprehensive overview of the landscape, knowledge gaps and future directions. Johns Hopkins University, 2016.

59

Mott MacDonald | Mapping and Assessing

Classified as Internal

companies provide collective coverage of every city, town and hamlet – far higher than neighbouring countries, including Ethiopia. 48% of the population report to have some form of mobile phone52, while three-fifths of the population are nomadic pastoralists or agropastoralists.

Decades of conflict and recent uprisings have taken a heavy toll and health systems remain extremely weak. It is also a crowded donor space with aid directed separately to Somalia and Somaliland as well as across both regions which also hampers coordination efforts. Key informant interviews with both Somaliland and Puntland ministries of health referred to weak coordination with and among development partners.

Pre-service training is the purview of teaching institutions (government or private), while inservice training and continuing professional development is provided mainly with the support of implementing partners, the main ones being Save the Children, World Vision, Care, Somali Red Crescent Society (SRCS) and UNICEF, who all have budgets to provide training. A point of feedback from providers reported by Puntland MoH was that the language of instruction and some materials were in English and were not well understood.

Depending on the topic, training might consist of classroom training (lectures, role plays) followed by 2-3 days practical training in hospitals, with training led by doctors or other experts.

During COVID, for first time the Puntland MoH provided training – in nutrition and COVID-19 case management - online via a zoom call, training over 500 participants this way. They reported, however, not having the means to do any assessment of this training and a concern that delivering remote training might compromise quality as they are not able to check what people have understood.

Community level training is generally not at all harmonised, with different CHW curricula in use.

Integrated Community Case Management (ICCM) is implemented but not well institutionalised.

The new cadre of Female Health Workers (FHWs), however, is starting to become better institutionalised because of strong Gavi support and more recent World Bank support to continue. FHWs undergo a 3-months pre-training.

Within both Puntland and Somaliland MoHs, there was some general awareness but not much ownership of some of the ongoing donor funded interventions. While government respondents were not familiar with specific films that have been developed, they agreed that the medium of film, or pre-recorded lessons, was useful, especially on topics such as EPI, cold chain management and that it was important they should be in the Somali language. For EPI, 60-70% of workforce are young graduates and turnover is high. Puntland MoH reported skill gaps (injecting techniques, stock management, reporting) as well as missed opportunities, such as screening child outpatient visits for immunisation history or nutrition when they come for other reasons.

Example Interventions Intervention

Description

HNQIS

Implementer: PSI, through the SHINE programme

Health Network Quality Improvement System

Detail: An electronic, tablet-based application used to improve quality of health services and monitor compliance to national guidelines. It covers 11 health areas, over four modules that support healthcare supervisors to: (1) plan supervision visits, (2) assess providers’ quality of care against clinical standards, (3) improve providers’ quality of care through tailored feedback, and (4) monitor quality

52 www.datareportal.com, January 2020

60

Mott MacDonald | Mapping and Assessing

Classified as Internal

Scale: PSI only operate in 3 regions within Jubaland, Puntland and Somaliland so their footprint is quite limited. All data are currently on a PSI server although plans in place with UNICEF and University of Oslo to adapt DHIS2 to support HNQIS. There is an opportunity to extend HQNIS to immunisation and to extend to private sector / urban immunisation strategy ODK

Detail: ODK is a free open-source suite of tools that allows data collection using Android mobile devices and data submission to an online server, without an internet connection or mobile carrier service at the time of data collection. In Somalia they have developed a COVID-19 application where contact tracing data from 4,000 healthcare workers (trained by the WHO) flows through ODK.

COVID –19 vaccine preparedness animation

Implementer: Medical Aid Films Detail: Animation on vaccine preparedness https://vimeo.com/477496771/fc301276e9. Developed with LSHTM, World Vision and a panel of reviewers, now in Phase 3 – engaging with Ministries.

Commissioned, as one of 12 films, by Digital Classroom, a group of mostly USbased e-learning organisations led by Last Mile Health. Already have successfully embedded animations into nursing and midwifery curricula in Somaliland.

Technological Uptake Despite good mobile service provision in Somalia, supervision of any new technologies in health facilities is very challenging as it is hampered by time, distance, logistical and budget constraints. As part of the SHINE programme, PSI’s HNQIS supports the use of technology for quarterly joint E-supportive supervision. While this has helped District Medical Officers and implementing partners work together to improve facility performance, it has been rolled out to limited locations so far.

Medical Aid Films report an enthusiastic response to the blended curriculum for nurses and midwives that they have developed in partnership with the government of Somaliland. The subsequent COVID-19 animation is at final stages and reported to be well supported by MoH. Somalia (and Somaliland) uses DHIS2 to track aggregate data, and DHIS2 Tracker for individual data in community or facility setting. The DHIS2 Tracker has been built to work seamlessly with DHIS2, and as part of that they have also deployed COVID-19 specific applications.

Governance Federal versus regional autonomy inhibits smooth rollout of initiatives and delays decision making. The Somaliland MoH and to a lesser extent the Federal MoH are relatively well engaged and working with donors on, for example, DHIS2. However there is a lack of capacity for effective DP coordination.

The Puntland MoH reported four key areas in which they would like support: Training for mid-level managers - while implementing partners train health workers, there is a lack of management training or mentoring for mid-level managers – the last time anything was provided was in 2015.

Database to harmonise training plans – it is “overwhelming” to monitor and keep track of what training is going on by implementing partners. MoH does not even have a database where they can track who is being trained and see any overlaps.

Skills analysis - to identify what competencies exist and where there are gaps

61

Mott MacDonald | Mapping and Assessing

Classified as Internal

Supervision system – while there is a nominal supervision system, there is no mechanism to provide on the job training. The nurse in-charge at a facility is also the person, for example, administering injections and there is no one able to verify if she is doing this correctly or not.

The Puntland MoH is motivated to ensure quality of care and would like better ways to measure impact of training and have feedback on how services are being performed. Somaliland reported similar issues, with better systems to organise their HRH data a priority.

The sustained and recent worsening of conflict in the region hampers all activities.

Sustainability HNQIS offer potential for digital assistance in quality monitoring and support, but the application is restricted by two factors: first it is only operating in 3 out of 18 regions, and second it holds data on PSI servers rather than a compatible application to the local DHIS2. The latter issue is currently being addressed with the help of the University of Oslo.

UNICEF are reported by various NGO implementers to resist any private sector involvement in immunisation.

5.4

Rwanda

Overview The Rwandan government has taken a strong lead on digital health and is involved in all initiatives, as it is with development work more generally. Through a Partnerships Unit they are able to convene, decide and communicate what the government wants to do and how.

Rwanda was one of the first countries to implement a national health information exchange (HIE) and whilst this was dormant for several years, the country is devising plans to revive certain aspects as part of the Healthcare Digital Transformation Strategy. The development of the HIE facilitated coordination of systems and reduced many silos, but the health information system still suffers from fragmentation, owing to a lack of data exchange standards and terminologies and uncertain funding for digital health initiatives. Recommendations that have been made include establishing a national health informatics society or association as well as restarting the eHealth technical working group, bringing donors, partners and government together to agree a common vision for digital health in Rwanda. A new Digital Health Department has been established - previously work involved both the Ministries of Health and ICT. In 2019, the Ministry of Health (MOH) of Rwanda partnered with MEASURE Evaluation—a project funded by USAID and PEPFAR—to conduct an assessment of the interoperability and readiness of the country’s HMIS. With regards to technology Rwanda scored well. In 2020 Rwanda launched its Electronic Immunisation Registry (EIR) which will reduce costs and provide live data, improving efficiencies and decision making.

Rwanda has ambitions to create Africa’s first universal primary care service and become a world-leader in digital health. In February 2020, the government signed a 10 year contract with Babyl, a mobile based primary health care provider (known elsewhere as Babylon). Babyl launched in Rwanda with BMGF support, and within 18 months, by April 2018, had registered 30% of adults. The aim now is to roll out Babyl to all Rwandans over the age of 12 through the government’s community-based health insurance scheme, Mutuelle de Santé.

62

Mott MacDonald | Mapping and Assessing

Classified as Internal

73% of the population currently report to have some form of mobile phone53 and 14.6% reported to have a smartphone54.

It was a Rwandan software engineering firm who developed the RapidPro open source platform for UNICEF. Having already adapted UNICEF’s RapidSMS platform to create TextIT, they then went on to partner with UNICEF to develop what became RapidPro. Example Interventions Intervention

Description

Babyl

As set out above the aim is to provide primary health care appointments, prescriptions and booking lab tests via a mobile phone. Babyl has signed an agreement with RSSB, the largest national insurance company and patients can access their prescriptions and lab tests using their MUTUELLE and RSSB (ex RAMA) insurance cards.

Babyl’s AI chat-bot enables non-healthcare trained providers and triage nurses to improve diagnostic capabilities and signposting to appropriate care more cost effectively.

Mental Health Issues

Implementer: Viamo with Partners in Health, Ministry of Health/Rwanda Biomedical Center. Funding from Johnson & Johnson.

Detail: Remote training solution – via audio lessons focused on mental health.

Collaboration with non-profit Partners in Health (PIH) to bring needed services to two districts in the Eastern Province and build a new generation of mental healthcare professionals by having psychiatric nurses from district hospitals train nurses in local health centres to diagnose and treat mental health. Delivered via a series of five-minute, pre-recorded voice calls on such topics.

Scale: 55,000 community health workers. Previously the government could only reach 25,000 workers, traveling from village to village for several years. Not only have patients seen improvement in their symptoms, but there has also been an economic impact - patients have missed fewer workdays, enabling them to better contribute to their families and communities. The platform had previously been used in the Ebola crisis and is now being used in the COVID pandemic (over 1 million messages sent) so there is big potential for scaling.

E-Heza Data Solutions

Implementer: TIP Global Health with Ministry of Health. Funding from Grand Challenges Canada, Izumi Foundation, Johnson & Johnson and Save the Children Detail: Rwanda’s first point-of-care digital health record gives nurses and CHWs tools to adopt evidence-based clinical care protocols, provide high quality care and utilise real-time data trends to tailor health education and improve health care.

Serves as an effective end-user interface that integrates with national databases such as OpenMRS and DHIS2 or serve as a stand-alone health record.

Technological uptake Babyl shows the extent of technological uptake when solutions are perceived to have value.

However, Babyl does not deliver preventive health care including immunisation.

53 www.datareportal.com, January 2020 54 Ministry of ICT, 2020 - Rwanda aims to collect 1M smartphones for poor families

63

Mott MacDonald | Mapping and Assessing

Classified as Internal

Viamo’s IVR activities have been delivered at scale, more than doubling the number of CHWs previously reached.

Governance The strong government interest and leadership ensure good levels of governance and country ownership. However, the top-down structure of government, with decision making power concentrated at senior level, can mean that progress on decisions / new innovations takes a long time.

Sustainability Babyl would appear to be sustainable. The Viamo mental health pilot programme does not complete until 2023 and assuming continued positive results the aim is that it is scaled to other countries in sub-Saharan Africa. Now that the platform is in place, the Rwandan government will be able to use it for other kinds of remote healthcare training. It is not clear, however, if this would require continued input from Viamo and funding from external sources / donors.

The E-Heza project has strong elements of sustainability and scalability hard wired into their approach as they are working directly with the Rwanda MoH to integrate into their national datareporting systems (DHIS2 and OpenMRS) while adding additional functions to the product. The aim is to bring E-Heza to all health centres and CHWs in Rwanda by 2021 and all of East Africa by 2023. While externally funded, critical to the success and long-term sustainability has been the Rwandan government investing people and time and being central to its development.

5.5

DRC

Overview Historically, digital health has been very fragmented, built organically, without coordination or leadership. To address this, the MoH is proposing the establishment of a decentralised public agency, Agence Nationale d’Ingénierie Clinique, de l’Information et d’Informatique de Santé (ANICiiS) to consolidate and drive forward all things related to digital health, health informatics and mHealth.

The Government of the new elected President is committed to improve immunisation coverage across the country. A comprehensive plan called Mashako (the country’s Emergency Plan for the Revitalisation of Routine Immunisation) was adopted in 2019, which is being used as a coordination platform and tool for mobilising additional support to immunisation. Digitalisation of all sectors, including health is now government policy. The recently created Digital Agency is the main institutional driver in this area. 40% of the population report to have some form of mobile phone55 (phone sharing also common). There is no reliable data on smart phone penetration.

Example Interventions Intervention

Description

‘Audio job aides’ and ‘Remote mobile based health worker training’

Implementer: Viamo working with IMA (UK funded) in Kinshasa and with Village Reach to reach 15,000 in 9 provinces (US funded)

Detail: 1. Audio job aides, uses IVR – 2-3 minute recordings on topics. Works by a “pull” mechanism, available on demand. Users dial a short code (42502) and hear message.

55 www.datareportal.com, January 2020

64

Mott MacDonald | Mapping and Assessing

Classified as Internal

2. Remote training - 8 module training on COVID; 5-6 minute recordings with 1 minute quiz at end. These are “pushed” to health workers. Village Reach collects mobile numbers of health workers.

Scale: 1. Audio job aides - up to 10 messages a month are free to Vodacom users, 11th message onwards charged at reduced rate.

2. Remote training - calls are free of cost to recipient (paid by external funder, negotiated contract with mobile operator including a PPP transition plan designed to make possible the continuation of the scheme under domestic funding (although unlikely to happen in DRC).

Appli Gestion PEV RD and Outil Suivi PEV RDC

Implementer: Acasus with in partnership with Immunisation Academy with funding from Gavi, BMGF, UNICEF. Detail: 1. Appli Gestion PEV RD was developed as part of the support provided by Gavi to the MASHAKO Plan. Tool to be used by the supervisors of EPI activities at Health Zone level.

2. Outil Suivi PEV RDC is a micro-planification tool developed by GRID3 (GeoReferenced Infrastructure and Demographic Data for Development) which provides maps with the estimated number of people to be vaccinated per village.

Vaccinators can also use it for reporting their activities. The combination of these two elements will enable a real time situation of the immunisation coverage per antigen.

3. Immunisation Academy (IA) App has 52 videos that will be forwarded to vaccinators (one per week for one year). Each video is 5 mins + quiz (5 questions).

Internet is needed to download the videos, but they can be seen offline. IA App is free of charge and the project provides the smartphones.

Scale: The interventions are planned to be scaled up to the whole country. Started in 3 provinces in 2019; 21 provinces are currently covered; all 26 provinces should be covered by the end of January 2021.

There are various other interventions funded, mainly by the US Government (under PEPFAR, M-RITE and Digital Square) and the BMGF implemented by PATH. Technological uptake Interactive Voice Response (IVR) is simple and easy to use. There is a low pick up rate for “push” calls can be as low as 30-40% in DRC but where supervisors have really got behind the training, response rates rose to 70%, showing its potential. Although calls free, the user needs to have credit on the phone to receive calls.

With the Appli Gestion PEV RDC intervention, supervisors receive a smartphone + SIM with data and enter data (offline) and send these through the system (health zone ->province>central level) when they are Internet connected. The data are managed centrally and used to populate an EPI Dashboard.

There is good ownership of regular mobile phones and reasonable 2G coverage although internet access is still an issue in rural areas. CHWs do not all have a smartphone, so they usually rely on devices provided by the projects. One major constraint is that per diems tend to represent a significant part of the health workers’ incomes so it is a difficult cultural change to move from face-to-face training.

Governance DRC is a huge country with a lack of staff at all levels of the health care system. Dealing with EPI at the central level is not enough as health programmes, including immunisation

65

Mott MacDonald | Mapping and Assessing

Classified as Internal

programmes, are increasingly managed at the provincial level (where technical and managerial capacities are very limited). Due to a somewhat dysfunctional but centralised system, some delays were reported in obtaining approvals as provinces imposed their own requirements.

Sustainability The initiatives largely rely on external funding and their scalability mostly depends on the will and capacity of the funding and implementing organisations to expand or continue support.

These initiatives are driven and funded by donors and external implementers so scalability and sustainability in terms of government ownership and funding are not really achievable in a context like DRC at the current time.

5.6

Senegal

Overview Senegal currently appears well placed to make the most out of digital health developments. It has a Strategic Digital Health Plan (PSSD) 2018-2023 in place, the aim of which is to drive and promote access to quality care, promote prevention and management of health risk.

In addition to this, the capacity, leadership and willingness of the EPI team in particular present an opportunity for the development of training and capacity programmes of vaccinators which fit with the actual needs and gaps.

Access to mobile phones is good with ownership equivalent to 109% of the population (suggesting some people have more than one) 56 and 34% reported to have a smartphone in 2018 57.

Example Interventions Intervention

Description

Coach2PEV and VacciForm

Implementer: Ganeshaid (as part of their Gavi HSS Grant to Senegal) Detail: Coach2PEV is a digital solution to measure EPI Performance Indicators and coaching immunisation workforce performance.

The tool can be used for several purposes: Collect data for surveys, E-supervision (Performance coaching function), training (VacciForm), a community of practices Scale: Currently being piloted (2018-2020) in some health districts, it should be scaled up during the next phase (2021-2022). Designed to be easily expanded: other countries (Benin in 2021) and other essential health programmes

Informed Push Model (IPM) or Yeksi-Naa (“I arrived” in the Wolof language)

Government of Senegal: Ministère de la Santé et de l’Action Sociale (MSAS) Ministry of Health / Pharmacie Nationale d’Approvisionnement (PNA) - National Supply Pharmacy Implementer: IntraHealth International, Dimagi Detail: Informs a push model of supply chain management in targeted health facilities, focusing on reducing contraceptive stockouts. A promising pilot led to a three-year expansion of IPM to all public health facilities nationwide.

56 www.datareportal.com, January 2020 57 2017, Basic mobile phones more common than smartphones in sub-Saharan Africa | Pew Research Center

66

Mott MacDonald | Mapping and Assessing

Classified as Internal

The system is tracking real-time facility-level data and making them available through a series of costumed dashboards at the district, regional, and national levels —allowing for improved supply forecasting.

Scale: Nearly 1,400 facilities. Extended to 118 essential drugs.

Delivers more than 100 essential health products to facilities and reduced contraceptive stockouts to an average of 2 percent across all health facilities, improving access to family planning for an estimated 3.2 million women.

There have also been a number of other solutions aimed at training health staff have been developed and tested by a range of implementing partners - AMREF (‘Cellal e Kisal’, ‘PRECIS’ and ‘JIBU’) and RAES. Sustainability The GaneshAid Coach2PEV work was developed with government (the MoH IT Department) and is relatively low cost (≈158K$, 2018-2020) – indicating good potential to be both sustainable and scalable. It is embedded into the domestic systems with planned migration to the country MoH server; capacity building for national leadership; and linkages with training institutions.

Outcomes of the pilot phase should provide useful lessons for the development of similar interventions, taking advantage of the improving connectivity and the appetite of the Senegalese population for new technologies. There is currently an evaluation of the pilot currently being conducted (February 2021) and will results will be available in due course.

The IPM / Yeksi-Naa programme developed by Dimagi and IntraHealth in close collaboration with MoH and National Supply Pharmacy, whilst not targeted at the direct learning and performance management of health workers, is a good example of government ownership and input into a successful nationwide intervention (that has been scaled over time) working with private and NGO partners. It also uses Dimagi’s CommCare platform.

However, whilst these interventions show the potential of what is possible, many others appear to face limitations and constraints: Lack of support from the Government making digital solutions not really sustainable (in the sense they will still be dependent on external funding at least) even when they are aligned to national strategies.

Financial barriers to end-users – training is now always free to end-users.

Internet access is still an issue in rural areas (even 2G is not available everywhere).

Limited engagement and support from mobile operators (which would help with cost and accessibility)

5.7

Indonesia

Overview Indonesia enacted the National eHealth Strategy in 2017. Despite this, and the fact that in recent years the telehealth and telemedicine industry has expanded in Indonesia, the use of digital innovation for the training and performance needs of its health workforce has not been maximised. Given its unique geographical composition, with numerous remote islands, and its large population size, further use of technology could add real value as the country transitions out of Gavi support. With its high levels of mobile connectivity, infrastructure and exposure to technology, it is well placed to do this, with mobile phone ownership equivalent to 124% of the population and 31.1% reported to have a smartphone in 2019. As detailed below, a number of

67

Mott MacDonald | Mapping and Assessing

Classified as Internal

interoperable global goods are already being used in Indonesia and that, along with consolidated government leadership, mean that it is well placed to take advantage of the benefits of digital.

COVID-19 has served as a catalyst in accelerating digital solutions and effective collaboration to deliver them. UNICEF has been central to this with three members of staff embedded in the MoH for the last year working on a range of digital communications and training focused on dispelling myths and ensuring health workers are appropriately trained in providing the vaccine – all with strong government leadership.

The WHO South East Asia Regional Office (SEARO) has identified some important lessons via feedback from the early introduction of the COVID-19 vaccine in Indonesia. First, the large-scale cascaded training and dry runs seem to have been key for logistical preparation and community confidence-building; and second, the innovative digital platforms used were adapted by the Government from existing platforms already used in country (e.g. Gavi-supported eVIN platform).’

Interventions Community Health Toolkit (Medic Mobile) The Community Health Toolkit has been deployed in one district health office in the Timor Tengah Selatan District and the Universitas Indonesia’s School of Medicine on an antenatal care coordination and gestational diabetes management project. Health care workers use the platform to manage the screening of gestational diabetes during pregnancy and after delivery, refer women with gestational diabetes for follow-up, and report on birth outcomes and gestational diabetes post-delivery.

CommCare (Dimagi) The Commcare platform has been deployed in Indonesia for some time. In 2014 World Vision, in cooperation with the MoH, used CommCare to develop an application integrated with an existing application, MOTECH Suite, for Integrated Health Posts (Posyandu) service and monitoring. This app meant that Posyandu staff were no longer required to write children’s health scores down manually in a book (which was regarded as a waste of time since nothing was done with this data) but instead, using a smartphone, they sent this data directly to midwives - their supervisors - through the online application who could download and analyse the data.

A number of key stakeholders were involved (MoH; Smart City; Institute of Development Studies UK; Dimagi; Grameen Foundation; Woman Welfare Association) and funding was provided by World Vision Canada and Indonesia as well as HSBC Bank. The programme ran from April 2013 - December 2017 and involved 220 Community Health Workers and Health Facility Workers and reached 11,300 children under 5. An evaluation concluded that cadres who used the mobile phones were more likely to provide feedback, accelerate the procedure of nutrition data collection, and improve data accuracy by 80%. Despite this success it is not clear if this programme continued once this phase came to an end.

RapidPro UNICEF has led a mobile health pilot aimed at boosting immunisation in urban Java utilising the RapidPro open source platform. A key principle underpinning RapidPro is that it both complements and integrates with national health systems – improving efficiency and ensuring both sustainability and country ownership. For instance, many of the digital health interventions powered by RapidPro have been linked into foundational digital health systems such as the DHIS2 and OpenMRS, resulting in easy user experiences for governments and reducing duplication and costs.

68

Mott MacDonald | Mapping and Assessing

Classified as Internal

For this pilot, local health workers go door-to-door to register newborns. Following that, automated reminders are sent to parents and guardians about upcoming vaccinations at local health posts (Posyandu). A separate feature allows health workers to record when vaccine stocks dip, permitting real-time monitoring of coverage by Government officials. Two years after the intervention’s launch, progress has been strong: nearly 3000 infants have been reached, and the programme has spread from Jakarta into dozens of urban districts in the neighbouring provinces of East and Central Java. All enrolled health facilities have provided monthly updates on the stock levels of each of the six vaccines monitored in the intervention.

Using the RapidPro platform, UNICEF has also been working closely with the Ministry of Health team for a national assessment of cold chain for COVID-19 vaccine distribution. It has also been used by health workers to feedback on their experience of training (60,000 trained) and ask about their experiences of the vaccine. About 10% of those trained (mainly nurses) have actively used it and the experience has been considered very positive and useful both for the health workers as well as policy makers and programme managers.

OpenSRP OpenSRP is an open-source mobile health platform that allows frontline healthcare workers to electronically register and track the health of their entire client population. It also provides programme managers and policy makers with data for decision-making. The COVID-19 track and trace application of OpenSRP is currently being tested and deployed in Indonesia with 200 healthcare workers but is too early for any results.

Viamo Viamo has been working with UNICEF Indonesia to develop an interactive voice response (IVR) mobile technology intervention to support the remote capacity building of 4,981 health workers on COVID-19 prevention and response in Papua and West. Viamo delivered three training curricula comprising 17 modules, on: Risk, Communication, and Community Engagement (RCCE); Infection, Prevention, and Control (IPC); and Safe Health Facilities. It was delivered in Papuan native language with calls lasting up to a maximum of 5 minutes. Included were preand post-tests were conducted to see the increased knowledge. The results data along with quiz results, metadata, number of calls, and completed lessons, were made available to UNICEF and partners via a hyperlink that enables the results to be downloaded as a CSV file for further analysis.

5.8

Summary

All the countries assessed are embracing digital health in some form and are developing capacity within government to better coordinate digital health activities going forward.

It is clear that while some interventions have been adopted at scale, others are more stand alone, relatively small scale, in isolated geographical areas, with no clear route to sustainability.

It seems to be common for even the most effective and popular interventions to struggle in terms of sustainability once donor support comes to an end with no plan in place for the longer term. In countries such as Ethiopia and Rwanda where the government takes a strong lead in setting the agenda of what they want (formalising this through plans and strategies) and coordinating partners accordingly, the risk of this happening is far less.

The challenge appears to be three-fold: firstly, there might be no overall landscape analysis at a country level of what is needed; secondly, there is often a lack of coordination in ensuring the alignment of initiatives that do show potential when scaling up; and thirdly, it is rare for initiatives to have a longer term sustainability plan in place. In some settings, such as Somalia, the lack of coordination mechanism between development partners becomes an aid effectiveness issue.

69

Mott MacDonald | Mapping and Assessing

Classified as Internal

Many interventions by their nature are innovative and dynamic and it is important that the creative space to be able to do this in a responsive and timely manner is not squashed.

Responding urgently to a crisis such as COVID requires swift action and thorough analysis and gaining buy-in from multiple stakeholders might be unrealistic.

Nevertheless, donors and implementing partners have an important role to play in helping ensure solutions align as far as possible to a coherent national plan - particularly in settings where capacity and digital maturity is less advanced. This means ensuring an aligned approach, investing in interoperable systems and ensuring that new innovations are compatible with the existing digital infrastructure.

There are a number of more commonly used ‘global goods’ that exist that have received significant and sustained funding from USAID, BMGF, Gavi and the Rockefeller Foundation.

The likes of Dimagi’s CommCare, Medic Mobile’s Community Health Toolkit, OPENSRP, DHIS2 Tracker have become widely used open source platforms, offer good levels of interoperability and are easily adapted to various needs and situations – such as responding to COVID quickly.

If countries are able to reach the point where their national HMIS systems are built on platforms stemming from such global goods, it reinforces an expectation that any new intervention, by default, must seamlessly integrate also. Ethiopia and Rwanda, with their strong leadership and drive on harnessing digital technology, are good examples. Interventions such E-Heza Data Solutions in Rwanda and the eCHIS in Ethiopia are interoperable with DHIS2 and OpenMRS. While there is limited evidence of initiatives being taken on and funded by government, there has been progress and it is now more common for interventions to be taken on by other donors when one funding cycle ends, so at least the intervention (if successful) continues. Critically, as examples in Ethiopia (COVID app) and Senegal (GaneshAid) have shown, governments increasingly progress to running interventions using their own systems, on their own servers, even if not yet fully funding it.

As countries increasingly develop national digital health strategies with associated costed plans, budgets should help to streamline funding for a core set of digital tools. As tools are rationalised, funding these tools could be more easily distributed across donors and increasing levels of support from country governments with the aim of full ownership and funding over time. The leap to full country funding is unlikely in most circumstances and some recurrent funding is needed. The speed at which digitalisation is progressing is significantly faster than that of progress on health systems strengthening, so limitations on digital rollout will become increasingly related to health systems weaknesses.

Considerations and feasibility in fragile and very low resources settings When it comes to frontline health workers’ performance management needs, fragile and extremely low-income countries are not considerably different from the rest of the Gavi eligible countries. As in most LMICs, both the number and distribution of properly trained health staffs are inadequate. The initial training received is usually of poor quality, working conditions are difficult (they have an important workload, operate in highly dysfunctional facilities as a result of weak health systems, receive minimal wages and sometimes need to dedicate part of their time doing a second job) and they often benefit from no or limited supervision and support. Also, many global platforms and open source digital tools cover and/or are used in fragile states, both in Africa and Asia. And the constraints and challenges training and performance management interventions face in fragile settings are not really different from those observed in more stable countries or at least in most of the rural or remote areas in these countries.

The difference between these settings and the better off and/or more stable ones is thus not a difference of nature, it is a difference in degree. The needs, constraints and challenges tend to be exacerbated: capacity gaps are deeper, health systems are even more dysfunctional or

70

Mott MacDonald | Mapping and Assessing

Classified as Internal

sometimes just non-existent, connectivity is more limited, health workers are more unlikely to be equipped with a smartphone, supervisions are more irregularly carried out.

Despite this overall similarity, settings in fragile settings may face additional difficulties, such as threats put on the health works in case of conflicts, extreme isolation when communications are affected by insecurity issues or when there is no road, outbreaks affecting mostly vulnerable populations (Cholera, Ebola, etc.), the presence of displaced populations who put an additional pressure on the health care delivery system, live in precarious housing or in camps and sometimes do not speak the same language. All these factors must be taken into account when designing training interventions.

Due to these possible specificities and the higher degree of constraints health care workers face in fragile settings, the balance between traditional face-to-face training and remote training options, namely through digital solutions, has to be reconsidered with more emphasis put on the remote approaches (using solutions that work with limited or no connectivity, no or minimal support from the peer and/or the supervisors). To be effective, digital solutions need to favour the simplest options, to be designed for trainees with sub-standard background and sporadic availability, to match with the languages used by the target populations.

Ideally, the content itself of the training should also be tailored in a way it provides the health workers the capacity to operate in their specific and unfavourable environment. Examples of factors that need to be reflected in the curriculum include: the special difficulties of the logistics (SCM, cold chain, power supply, etc), simplified schemes for collecting data and reporting and adequate support and supervision modalities.

The Figure below sets out the sort of interventions that could be appropriate based on the country context and maturity of digital health infrastructure. It is purely indicative and should not be treated as prescriptive.

71

Mott MacDonald | Mapping and Assessing

Classified as Internal An indicative summary of what works where:

72

Mott MacDonald | Mapping and Assessment

Classified as Internal

6 Donors/ Funders 6.1

Bilaterals

Bilaterals are all in the digital innovation space either as funders or as developers of digital programmes including learning and training and performance management. All have digital strategies which include health and cover learning and education. Most have signed up to the Principles for Digital Development. The key ones are USAID, DFAT and FCDO and Agence Française de Développement (AFD) UK Foreign, Commonwealth and Development Office (FCDO) FCDO is still in a transition phase from DfID and integration is still taking place between the former FCO Prosperity Fund -a separate UK Aid programme and DfID. However digital work in health is directed by DFID’s Digital Strategy 2018-2020 which sets out sets out a vision and approach for doing development in a digital world. It is supported by three main programmes The Prosperity Fund cross-HMG 'Digital Access Programme' is a DFID-led partnership with FCO and DCMS with a budget of £105 million. It aims to catalyse more inclusive, affordable, safe and secure digital access for excluded and underserved communities in Kenya, Nigeria, South Africa, Brazil and Indonesia.

The Educational Technology Hub’s plan is to accelerate progress in developing world education by 2030 through effective use of digital in education systems. This includes the use of digital, data and technology in schools, in ministries and at home to improve learning, efficiency and value for money. It currently provides innovative support, research on learning outcomes, and direct support for governments and global leadership.

Unlocking Digital Impact for Development is a programme to help to deliver the strategic goals of DFID’s Digital Strategy. The programme budget is £18.5 million between 2019-20 to 2023-24.

Delivery will primarily be through the Digital Impact Alliance, housed within the UN Foundation.

There is a separate pillar to finance complementary policy research and advice activities with other partners, including Digital Pathways at Oxford, as well as a learning and evaluation pillar.

To date there are no specific digital projects aimed at front line health workers. However, work may be going on at country level as part of more general programmes.

These three have been complemented by the Frontier Technologies hub which is probably of most interest to Gavi as it also hosts digital work to support the COVID response in LIMCS through its COVID Action programme. It works across three main areas: Livestreaming explores the use of frontier technology by working with partners all over the world to test and scale tech with the potential for positive social impact. Futures connects FCDO people with one another and the world of tech, equipping them to apply frontier technologies in their programmes. Hub Research gathers & shares what we learn and dives deeper into areas where tech has the greatest potential for doing good. While the website encourages posting by developers and innovators and shows multiple examples of good digital practice in health systems strengthening there is nothing to date on training g or performance management of frontline health workers. The COVID action section again has good examples of useful technologies but nothing on learning or training or performance management.

FCDO also funds the Digital Impact Alliance (DIAL) (digitalimpactalliance.org) DIAL’s work focuses on streamlining technology, unlocking markets and accelerating the rate at which others can deploy digitally enabled services. It could offer support to developing digital platforms as part of country digital strategies.

Australian Department for Aid and Trade (DFAT) 73

Mott MacDonald | Mapping and Assessment

Classified as Internal

The most relevant programme for Gavi is the use of radio for training backed up by SMS messaging aimed at supporting front line health workers for the COVID response. The programme is run by UNICEFs across 14 countries of the Pacific. The training will be aired over a period of six months with a total of 33 broadcasts of 30-minutes each that aim to support governments to connect practicing nurses and midwives with the opportunity to learn, share information, and incorporate new WHO guidance on COVID-19.Participating front-line healthcare workers will also have the opportunity to ask questions for future episodes, share learning needs and receive episode summaries through UNICEF’s RapidPro platform, which is a two-way communication system that works with free SMS and messaging apps from smartphones.

The impact of COVID-19 on travelling led to a rapid transition to distance learning in DFAT aimed at nurses supported programmes in the Pacific which it outsourced to Medcast an Australian provider. They are a major provider of online curses aimed at medical and nurse practitioners in Australia New Zealand and the Pacific and have a huge menu of online webinars and courses some free some to be paid for None specifically cover vaccinations but they would have the capacity to provided them.

DFAT also supports the on-line training of veterinary professionals through The Asia Pacific Consortium of Veterinary Epidemiology led by the University of Sydney and para vets through a program lead by Charles Sturt University will develop online training for para-vets in the Pacific.

Both would have the capacity to draw up similar programmes for front line health workers.

USAID Like other bilaterals USAID has a digital strategy - A vison for action in global health. Its focus is on addressing the fragmentation in the digital landscape and building up national capacity. This is covered in three of its four priorities. The fourth is leveraging global goods such as opensource platforms

One of the most useful tools they have produced is The Global Digital Health Index. The GDHI is an interactive digital resource that enables countries to assess their maturity in digital health and benchmark themselves against other countries. the GDHI was prepared using the WHO/ITU eHealth Strategy Toolkit and used its design process to engage with different data

74

Mott MacDonald | Mapping and Assessment

Classified as Internal

producers and data users. The index provides a quick snapshot of digital health ecosystem maturity at the global and national levels that can be easily shown at presentations and events.

It complements the WHO digital health atlas.

Agence Française de Développement (AFD) AFD runs learning and training programmes through the AFD Campus (ex-Cefeb). IT has a significant digital training component which continues to expand. Currently it has free online courses open to all ("Massive Open Online Courses", in English, or MOOC), training tutored remotely, virtual classes, webinars, serious games, communities of experts, etc. In the end, each AFD partner who wishes to strengthen their knowledge can build an à la carte course.

Courses are in French. Of particular value to Gavi might be the project management and leadership courses.

AFD has also has made a major commitment to support the WHO Academy. Earlier this year, WHO and France signed a Declaration of Intent to reach millions of people with innovative learning via a digital learning platform, with a hub in Lyon and embedded in all six WHO regions.

6.2

Foundations/ Innovation Funders

Bill and Melinda Gates Foundation (BMGF) BMGF has been at the forefront in terms of digital health, funding the development of platforms as well as more specific programmes and interventions. Their approach and funding structure means that they have more creative freedom in what and how they fund. Their strategy on innovative technology solutions has the very purposeful goal, ‘to identify emerging technologies that have potentially transformative applications for global health’. In 2017, as part of their Goalkeepers initiative, Gates set the following ‘accelerator’: Scaling national community health worker programs. Their strategic interest in both innovative technology and scaling and empowering frontline / community health workers means they have been a key stakeholder and direct funder of many of the initiatives and interventions that this work has explored. Below are some examples (with fuller details in the grid and country case study sections) •

• • • • • • • •

Last Mile Health and their Community Health Academy work (Liberia, Uganda, Ethiopia). Related to this they built the Global Faculty Network, a network of global experts to provide oversight and guidance to the Academy team.

BBC Media Action’s Mobile Academy in India Bull City Learning and their Immunisation Academy Watch work (global) The Malaria Consortium’s inSCALE work in Mozabique and Uganda Appli Gestion PEV and Outil Suivi PEV in DRC - training of vaccinators through Immunisation Academy, led by Acasus OpenSRP - an open-source mobile health platform that allows frontline healthcare workers to electronically register and track the health of their entire client population Better Immunisation Data programme run by PATH in Tanzania and Zambia Village Reach - Access to Health Programs in Rural Africa Dimagi – various initiatives using their CommCare platform

Rockefeller Foundation The Rockefeller Foundation is heavily involved in the funding of digital health initiatives. In support of countries and those on the frontline in their COVID-19 responses, they are awarding grants totalling $3 million to four organizations: Dalberg, Dimagi, Medic Mobile, and Odyssey Energy Solutions.

Rockefeller is jointly funding Dimagi and Medic Mobile, the two largest developers of apps designed specifically for community health workers (CommCare and Community Health Toolkit

75

Mott MacDonald | Mapping and Assessment

Classified as Internal

respectively) to co-develop a suite of open-source digital solutions to aid countries’ efforts to contain, mitigate, and recover from COVID-19 more efficiently. The aim of the grant is to enable Dimagi and Medic Mobile to engage a wider community of organizations in sharing resources, tools, and lessons learned. The hope is that this collaboration will facilitate the creation of more integrated digital health technologies that can support both the current COVID-19 pandemic response, and address future and existing health challenges.

Dalberg has received a grant to strengthen the Incident Management System (IMS) capacity of West African Emergency Operations Centers to prepare for, detect, and respond to public health emergencies. The foundation’s grant will support work in six West African countries: Senegal, The Gambia, Guinea, Guinea-Bissau, Mali, and Mauritania.

Odyssey Energy Solutions is receiving funding to develop its data platform, which will enable the fast and sustainable deployment of donor capital to energize healthcare facilities with distributed renewable energy technology. The Odyssey platform will align donor efforts, targeting efficient allocation of over $200 million across at least 2,000 health centres in subSaharan Africa.

Rockefeller is also supporting Gavi directly, providing a US$ 5 million investment committed to strengthening the critical role frontline health workers play in delivering immunisation services in Gavi-supported countries through digital tools and innovative information-sharing approaches that help them improve equitable access to life-saving vaccines. The aim is to aims to support the rapid implementation of innovative solutions and how best to use digital technology, leverage new partners, generate insights, and build a strategy to improve health workforce development and performance in Gavi-supported countries.

Johnson and Johnson Foundation J&J have a keen interest in supporting work that improves the effectiveness of health care workers in developing countries with a focus on the potential of digital interventions. For instance, in Rwanda they are funding Viamo to provide telephonic messages including gaming scenarios to help health care providers deliver better care related to mental health. In Rwanda, they have also provided support to E-Heza Data Solutions, Rwanda’s first point-of-care digital health record system that gives Nurses and Community Health Workers (CHWs) the tools they need to adopt evidence-based clinical care protocols, provide high quality care and utilize realtime data trends to both tailor health education to individual family needs and to improve the health care delivery system while simultaneously satisfying Rwanda Ministry of Health data reporting requirements.

In Sierra Leone they have also been involved in a public-private partnership, MOTS - Mobile Training & Support Service, that provides remote training using IVR Technology on mobile phones as a complement to classroom trainings, with modules on vaccination & outbreak response and disease surveillance.

76

Mott MacDonald | Mapping and Assessment

Classified as Internal

7 Sustainability 7.1

Sustainability Landscape

Digital learning and training and performance management need to be seen within the wider lens of digital innovation which faces the same challenges as any new intervention in global health. Sustainability is as always, the biggest. There is now a well-established literature on sustainability in development with clear guidelines for good practice to achieve it. Ultimately until a major intervention is embedded in a country’s relevant sector plan with funding provided by the government or by individuals it will not be sustainable. The exceptions are short term time limited interventions which can be financed until completion by a development partner. This can include training and education of individuals where in addition participants may self-finance if there is sufficient motivation. It will not apply to performance management systems which will rely on government uptake.

However, a quick landscape analysis shows that digital innovation is characterised by multiple projects funded for the short term by development agencies with much of the innovation taking place in OECD countries and delivered in LMICs by NGOS both for and not for profit with little evidence of sustainability This has been recognised for some time by the major development agencies and they have produced ‘The Principles for Digital Development’ (www.digitalprinciples.org) a set of nine guidelines for integrating best practices into technology-enabled development programs for international development and cooperation and endorsed by over 100 development organizations. The endorsement programme was led by USAID. The United Nations Foundation's Digital Impact Alliance is responsible for their use. Gavi is currently not one of the endorsers.

Sustainability is one of the key principles and the detailed guidance for the Principles sets out what is now generally accepted as good practice.

The nature of digital innovation is such that the innovators may be good at coming up with ideas and getting them to market but have little experience in working with governments for sustainability. Conversely staff in governments and the multi-lateral and bilateral development agencies may not be at the cutting edge of knowledge of digital development. Last year when the UK Aid programme had a competition for seed funding for digital innovations for resilient health systems including training there were 560 applications none of which were from governments in LMICs. However, WHO through its 2020-25 Global Digital Strategy is supporting governments in LMICS in developing country Digital Implementation Plans. If these take off and development partners at country level work through them then they will offer a roadmap to sustainability.

7.2

Global public goods and potential market shaping / funding models

An increasingly important concept in assessing digital solutions is Total Cost of Ownership (TCO) (for government) which is calculated through a comprehensive evaluation of all costs associated with digital solutions, including all expenses pertaining to hardware and software procurement, management and technical support, communications, training, system upkeep, updates, operating costs, networking, security, and licensing costs.

Another important concept is of Global Public Goods for health and how they are financed. In a post-pandemic world, upskilling health workers for greater effectiveness and vaccine coverage is undeniably a global good. Yet some of the largest technology companies maintain that the market for digital health global goods is functioning poorly at scale. To maintain service

77

Mott MacDonald | Mapping and Assessment

Classified as Internal

functionality, make continuous incremental improvements, and maintain data protection to digital platforms, there is a recurring cost.

International development assistance tends to happen in cycles – institutions receive new funding for solution development while donors avoid funding recurrent costs in an effort not to create dependency. This approach becomes inefficient, however, given that upfront development costs are high. There are numerous examples of INGOs from the global north building solutions and when the project ends, the goods are lost.

The question of sustainability is a question many countries and organisations are currently struggling with. This is particularly true of software systems with open source licences where the expectation often is that the solution is free when in fact the cost of the license is a very small component of the total cost of ownership of the solution and long term sustainability needs to be financed. Some countries have developed a digital health investment case to guide the development of digital health systems and motivate for funding from the national treasury for identified technologies and systems with proven healthcare benefits.

Ideally a market would exist beyond the initial project investment phase that would incentivise producers and implementers of Digital Health Global Goods (DHGG) to ensure maximum value of the platforms. However, there is currently little market for pricing the ongoing support and maintenance of the deployed global good past the front-loaded investment phase. Because of the lack of market, revenue models for providers of DHGG at scale are unproven and this leads to a lack of investment by tech companies and instability for government implementors.

Paradoxically, average costs might be as little as US$2 per (health worker) user per month and could fall as more users were added, allowing more value to be created.

Adding value to existing platforms of apps would make more sense than developing new apps and platforms. Providing bridging funding over a longer period to support ongoing costs until a transfer of full ownership to government, when value is better demonstrated at scale and investment cases made, would also be cost effective. Focusing on strengthening health systems, supporting and aligning with national digital as well as health workforce strategies, would make projects more complex but also more impactful in the long run.

The same issue arises as WHO contemplates how to finance its new Academy. A “Netflix” style subscription model is one possibility, wherein the Academy would need to continuously provide value or lose out to competitors. WHO did a market analysis in 40 countries on what people would be willing to pay for a WHO-branded learning experience. In high income countries (HIC), health professionals would pay in several hundred dollars (and would be willing to pay extra to help subsidise users in poorer countries) and there was willingness to pay in LMICs also. A differentiated subsidy model could allow cross-subsidies from HICs to LMICs. Gavi is a global leader in market shaping strategies in global health, for example, increasing the number of suppliers of the pentavalent DTP-HBV-Hib vaccine from 1 to 5, and reducing the price per dose from $3.50 to $0.68.58 It could consider leveraging its position to address the long term and intractable issue of sustaining Global Goods in the digital medical education space.

7.3

Actions Gavi can support

Gavi cannot be responsible for the sustainability of all digital innovations in learning and performance management at country level. It can however contribute by (a) following the guidelines set out above and signing up and adhering to the principles of digital development when introducing digital innovations at country level (b) making it clear to developers of digital training and performance management programmes that if they wish to be considered for Gavi 58

Financing Global Common Goods for Health: When the World is a Country. Gavin Yamey et al. Health Systems and Reform 5(4):334-349, 2019. WHO

78

Mott MacDonald | Mapping and Assessment

Classified as Internal

funding or support they must adhere to the Principles for Digital Development including those for sustainability.

If Gavi supports digital innovations in learning/training and performance management for relevant front line health workers at country level then it should consider doing it as part of its HSS work ensuring it integrates into country digital innovation plans where it can work with governments. The following are some examples of good or emerging practice that it could draw on.

Ensuring Gavi staff and relevant country vaccine programme staff have skills for digital development Futures is a programme from the Frontier Technology Hub (UK Aid’s innovation platform) that aims to develop FCDO in-country office staff’s knowledge, confidence and skills to use frontier technologies like drones, 3D printers and blockchain to deliver development objectives. It’s built on the concept of exploring ideas together without any predetermined answers, learning from multiple perspectives and hearing from technologists working on the ground who have tangible and relevant experiences to share. It has 25+ activities that FCDO departments can choose from to shape their Futures experience: everything from quick wins like methodology training, speeddating technologists and setting up a virtual Slack/WhatsApp community around the tech ecosystem, to high-investment, high-impact options like setting up a rolling makerspace or a start-up-FCDO secondment programme.

This is supported by Frontier technology live streaming which provides real time online support and partnering staff with digital experts.

Gavi could either set up or buy into a similar programme for its own staff and extend it to relevant vaccine programme staff at country level Working through the WHO Academy With $100 million funding from the French government the WHO Academy has been developed and is ready to launch in May 2021. There will be a hub in Lyon with six regional campuses. Its aim is to be a major learning centre for the global workforce. The Academy’s commitment to open source solutions will bring communities together to develop global public goods for digital learning, recognition of learner achievement, multilingual learning, learning technologies and innovations to ensure no one is left behind.

It may offer an opportunity for Gavi to work with partner governments to commission appropriate digital courses for front line health workers which would have legitimacy via accreditation and sustainability.

Moving from projects to programmes Gavi may well wish to introduce short or longer term digital training and learning programmes for front line health workers as well as longer term performance management programmes that have been developed and demonstrated by an NGO provider. It should do this in collaboration with government and involve them in any selection process. Ideally, they should be part of the National Digital Implementation plans being supported at country level as part of WHO’s global strategy on digital health. It may well be that Gavi HSS funds can be used as a bridge between the initial developer and the government but ideally longer term programmes should not be introduced until government approval is given and a MOU of transfer of future funding in place.

The guidelines for sustainability set out in the ‘Principles for Digital Development’ should be followed. They are: •

Plan for sustainability from the start.

•

Develop a definition of sustainability for your initiative.

•

Identify and implement a sustainable business model.

•

Use and invest in local information technology service providers.

79

Mott MacDonald | Mapping and Assessment

Classified as Internal

•

Engage local governments and integrate national strategies into programming.

•

Collaborate instead of competing, and partner to identify the best approach with the greatest impact.

•

Build a program that can be adapted as user needs and the context change.

•

As mentioned above the innovators and developers of digital training, learning and performance management programmes may well be unaware of the clear evidence

However, developers and innovators of digital learning and training programmes may not be aware of these so Gavi can play a role in making clear to developers who approach Gavi for funding or for support in implementing their programmes at country level that they need to follow these guidelines.

Market shaping and supporting interoperability between different systems Gavi has good relationships with governments and could use its political capital to help support market shaping for sustainable funding models of Global Goods. It could also use its influence to promote the development of ecosystems that support interoperability between different systems with tailorable app platforms – rather than development of new individual apps.

80

Mott MacDonald | Mapping and Assessment

Classified as Internal

8 Strategic Opportunities for Gavi Specific initiatives have been mapped and assessed against criteria of scalability, agility to adapt to rapid developments, potential for measurable impact, and country ownership and sustainability. The results of this exercise are provided in Annex 4.

WHO provided a useful summary of the big picture outlook, specifically on digital education for health workers, which is reproduced below:59

In taking forward support for strategies which accelerate health provider competency development and performance, Gavi should consider its unique role in supporting solutions to achieve scale and gain traction at country level and globally. The following actions are suggested: 1. Join forces with other multilateral organisations such as WHO, UNICEF, the World Bank and The Global Fund using their platforms to support improved global and in-country coordination of frontline health worker interventions. This could include: o Advocating for more internationally recognised, nationally accredited, usercentric learning experiences including via the WHO Academy in collaboration with local ministries, professional councils and associations; o Investing in courses for vaccination managers and further language translations of successful courses; o Supporting ministries of health in fragile settings establish databases for health provider training to avoid duplication; o Running coordinated live simulation exercises with in-country partners e.g. for pandemic preparedness / vaccine roll out.

2. In relation to digital learning and performance solutions, build on what exists and invest in further language translations, local adaptation and contextualisation, supporting interoperability and sustainable funding mechanisms for existing proven technological solutions; in particular, o Using blended, multi-faceted approaches rather than a single approach – opting for integrated digital solutions which address competency building, performance feedback, and accountability measures such as digital vaccination tracking; 59

From WHO Digital Education for Building Workforce Capacity - 2020

81

Mott MacDonald | Mapping and Assessment

Classified as Internal

o

Supporting initiatives such as Digital Classroom which can rapidly update, translate and roll out materials

o

Encouraging more partnerships between platforms such as CommCare and Community Health Toolkit in support of interoperability.

3. Ensure rollout of solutions for provider education / performance is country driven and aligned with wider support (through HSS grants) to address health workforce and other health systems constraints. This might include: o Providing technical assistance to countries to address staff turnover by developing career ladders for frontline health workers, with performance reward and recognition strategies, and investment cases for approaches to reduce wasteful turnover; o Ensuring in-service training strategies have MoH buy-in and there is a clear link to career development; investment in a network of local partners for sustainability; o Fostering greater links with pre-service training institutions, and providing technical support to digitise elements of pre-service curricula; o Consolidating and leveraging more value from existing platforms within countries, for greater economies of scale and to build familiarity with common platforms.

4. Ensure that all innovative/digital investments are aligned to the country digital health plan if one exists, digital development principles and country cMYPs and ensuring investments are aligned with and communicated to other development partners. o

Encourage co-financing from country government for implementation of digital technologies, and build up capacities of local Digitech companies through northsouth business partnerships.

5. Channel support to district level capacity building for quality improvement initiatives This could include: o

Identifying lagging behind districts and agreeing collaborative quality improvement initiatives - identification of the problem, root cause analysis, action planning, agreement on focused indicators, data collection and rapid feedback cycles;

o

Deploying a cadre of technically savvy graduates to be embedded within district teams to support digital elements within above initiatives, particularly troubleshooting use of digital data systems and producing data visualisations; Building capacity of governments for in-house data analytics and use of data; Supporting district-level management training, to build exposure to less hierarchical, problem-solving / coaching approaches.

o o

6. Seek to measure impact in relation to the original problem that is being addressed using better, fewer, metrics that reflect quality of service delivery. This might include: o

o

Ensuring regular monitoring of a limited number of indicators, rapid feedback cycles and two-way data flow that allows adaptation, layering and continual improvement of interventions; Using mobile phone surveys for collecting rapid feedback from health service users on quality of care, missed opportunities for integrated care etc.

7. Ensure better alignment of funding and incentives and consider innovative payment for performance mechanisms to replace training per diems, based on empirical evidence of service improvement.

8. Allow disruptive bottom up strategies that challenge traditional power dynamics, but explore the feasibility of pushing similar models down to lower tiers where non-

82

Mott MacDonald | Mapping and Assessment

Classified as Internal

English/French speaking participants can engage in their own language. This might include: o Exploring learning laboratory approaches to distil and continually refine such approaches; o Considering a financial mechanism to support “bottom-up” action plans developed through this route via HSS grants, for example micro grants which alumni of such networks could apply for (with involvement of district health teams) for small scale collaborative quality improvement initiatives.

83

Mott MacDonald | Mapping and Assessment

Classified as Internal

Annex 1 - Key Informants List #

Organisation

Person

Date

1

Viamo DRC

Meredith Baker

30-Nov

2

MSF

Natalie Roberts

30-Nov

3

SHINE

Wilma Meeus

27-Nov

4

MedicalAidFilms

Catherine McCarthy

30-Nov

5

Gavi (Pakistan)

Alexa Reynolds / Maria Ramirez / Riswana Soundardjee

01-Dec

6

Acasus

Ahmed Razzak

08-Dec

7

Amref Senegal

Sarah Razafindramanana

27-Nov

8

PATH DRC

Trad Hatton

07-Dec

9

RAES/Keewu Production

Alexandre Rideau

08-Dec

10

Community Health Academy

Magnus Conteh

08-Dec

11

GAVI (Senegal)

Antoinette Awaga, Jean-Charles Dubourg

09-Dec

12

Viamo Rwanda

Kennet Christiansen

08-Dec

13

AKU Pakistan

Dr.Momin Kazi and Dr.Hasan Nawaz

04-Dec

14

BMGF Pakistan

15

Gavi Pakistan M & E

Riswana Soundardjee

07-Dec

16

National EPI Pakistan

Dr.Naeem, Dr.Soofia Yunis,Aasim Zaki

04-Dec

17

Punjab EPI Pakistan

Bashir Ahmed

01-Dec

18

Sindh EPI Pakistan

Lala Aftab

03-Dec

19

UCL Knowledge Lab

Allison LittleJohn

20

Individual / author Lancet systematic review

Alexander Rowe

09-Dec

21

BMJ

Kieran Walsh

11-Dec

22

UNICEF Pakistan

Yasmine Choulby, Khalid Nawaz, Andrey

11-Dec

23

IRD Global

08-Dec

11-Dec Subhash Chandir, Danya Arif

24

Village Reach and Praekelt

Carla Blauvelt and Rebecca Alban – Village Reach; Chiara Kunnie Praekelt

02-Dec

25

Acasus DRC

Christelle Mputu, Amani Lwanzo and Oriane Gascon

11-Dec

26

Acasus Ethiopia

Kirobel Behashaw and Jonny Barty

10-Dec

27

Last Mile Health Ethiopia

Abraham Zerihun

14-Dec

28

BMGF - Seattle

Nicholas Leyton and Tove Ryman

16-Dec

84

Mott MacDonald | Mapping and Assessment

Classified as Internal

29

Dimagi

Jordan Lerner, Jonathan Jackson, Mohini Bhavsar

16-Dec

30

Laerdal Global Health

Unni Silkoset, Michael Sautter

17-Dec

31

Geneva Learning Foundation

Reda Sadki, Ian Steed

18-Dec

32

Medic Mobile

Isaac Holeman

18-Dec

33

PSI Somalia

Saba Khan

21-Dec

34

PATH India

Sandeep Kumar and Dr. Satyabaha

35

Centre for Distance Education

George Siemens

05-Jan

36

Learning Foundation / University of Georgia

Karen Watkins

05-Jan

37

USAID

Amy Lin, Rachel Wood, Malak Elshafei

05-Jan

38

THET

Louise McGrath

11-Jan

39

WHO

Jim Campbell

13-Jan

40

JSI Ethiopia

Jonathan Metzger, Hibret Tilahun, Dr. Loko Abraham

26-Jan

41

Digital Health Education Collaborative

Josip Carr

25-Jan

42

John Hopkins University CCP

Alice Payne Merritt; Sanjanthi Velu

29-Jan

43

Jembi South Africa

Chris Seebregts

02-Feb

44

Gavi Rwanda

Rachel Bent

10-Feb

45

JSI Ethiopia (embedded in MoH)

Amanuel Biru

10-Feb

46

MoH Somaliland

Bashir Ahmed

10-Feb

47

MoH Puntland

Edil Hassan, Rasaaq Abshir Xirsi, Abdirashid Omar Warsame

11-Feb

48

UNICEF Indonesia

12-Feb

49

Immunization Academy Watch/Bull City Learning

Riziky Siyafitri Nathan Pienkowski, Amanda Beal

50

RISE - JSI

Lora Shimp; Parthasarathi Ganguly

2-Mar

51

UNICEF HQ

Naina Ahuja, Alissa Collins Desiree Raquel Narvaez

23 Mar

22-Dec20

2-Mar

85

Mott MacDonald | Mapping and Assessment

Classified as Internal

Annex 2 - Bibliography Document reference High-quality health systems in the Sustainable Development Goals era: time for a revolution. Margaret E Kruk et al. The Lancet Global Health Commission, 2018 Digital education for building health workforce capacity. Geneva: World Health Organization; 2020. Licence: CC BY-NC-SA 3.0 IGO

Lancet Global Health Commission on High Quality Health Systems - M Kruk et al 2018.p

Digital Education for Building Workforce Capacity.pdf

Report of the consultation meeting on digital health interventions and health workforce capacity building, WHO

meeting_report_Digit al-health-interventions-health-workforce-capacity-budiling Copy.p

Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Alexander K Rowe, Samantha Y Rowe, David H Peters, Kathleen A Holloway*, John Chalker, Dennis RossDegnan. Lancet Glob Health 2018; 6: e1163–75. Published Online October 8, 2018

Lancet Rowe et al systematic review.pdf

The effectiveness of training strategies to improve health care provider practices in low- and middle-income countries. Rowe AK, et al. BMJ Global Health 2021;6:e003229. doi:10.1136/bmjgh2020-003229

Rowe_2021 (HCPPR training effectiveness) (article and appendix 1).pdf

Garcia-Elorrio E, Rowe SY, Teijeiro ME, Ciapponi A, Rowe AK (2019). The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: A systematic review and meta-analysis. PLoS ONE 14(10): e0221919

Garcia-Elorrio_2019 (SR on CI in LMICs).pdf

Mobile Technology in Support of Frontline Health Workers - A comprehensive overview of the landscape, knowledge gaps and future directions. Johns Hopkins University, 2016

JHU review mHealth for frontline 2016.pdf

Medical Aid Films, Impact Report 2016 - 2019 MAF Impact Report 2016-2019 (1).pdf

Msemo, G et al. Newborn Mortality and Fresh Stillbirth Rates in Tanzania After Helping Babies Breathe Training. Pediatrics. DOI: 10.1542/peds.2012-1795. 2013

hbs_article_tanzania. pdf

86

Mott MacDonald | Mapping and Assessment

Classified as Internal

Rapid Immunization Skill Enhancement (RISE) - John Snow International. Project documentation.

JSI BRICKS Case Studypdf.pdf

RISE BULLETIN_01 Jan 2021.pdf

Geneva Learning Foundation - Teach to Reach and Scholar Covid-19 Peer Hub. Project document.

GIN narrative Peer Hub.pdf

Coach2PEV and VacciForm – GaneshAid. Project document HSISmeeting_Ganesh AID_Final_8Dec2020.pptx

McKenna, P et al. Mobile training and support (MOTS) service— using technology to increase Ebola preparedness of remotelylocated community health workers (CHWs) in Sierra Leone. mHealth. 2019

Sierra Leone MOTS.pdf

Immunization Academy Watch. Efficacy Study, project document IA_Efficacy_Study.pdf

Transitioning Well: Building Sustainable Solutions. Village Reach, organisation document TransitioningWell_FI NAL.pdf

Viamo's remote training and audio job aides. Organisation documents Viamo Activities on COVID-19 Globally.pdf

Remote Training and Mobile Curriculum_Viamo.pdf

Case Study on the Health Network Quality Improvement System (HNQIS), USAID Annex-D_Case-Study -on-HNQIS Copy.pdf

Health Network Quality Improvement System (HNQIS), Review of Quality Improvement Systems. PSI project document 20161014-HNQIS-Lit -Review Copy.pdf

87

Mott MacDonald | Mapping and Assessment

Classified as Internal

Document reference

Weblink

Digital education for building health workforce capacity.

Geneva: World Health Organization; 2020. Licence: CC BY-NC-SA 3.0 IGO

https://www.who.int/publications/i/item /dfigital-education-for-building-healthworkforce-capacity-978-92-4-0000476

WHO. Digital Implementation Investment Guide (DIIG): Integrating Digital Interventions into Health Programmes.

Geneva: World Health Organization, 2020

https://www.who.int/publications/i/item /9789240010567

O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open 2018;8: e021467. doi:10.1136/bmjopen-2017-021467

https://bmjopen.bmj.com/content/8/4/e 021467

WHO. Working together for health. World Health Report 2006. Geneva: World Health Organization, 2006

https://www.who.int/whr/2006/en/

WHO. Digital Implementation Investment Guide (DIIG): Integrating Digital Interventions into Health Programmes.

Geneva: World Health Organization, 2020

https://www.who.int/publications/i/item /9789240010567

The effectiveness of training strategies to improve health care provider practices in low- and middle-income countries. Rowe AK, et al. BMJ Global Health 2021;6:e003229. doi:10.1136/bmjgh-2020-003229

https://gh.bmj.com/content/bmjgh/6/1/ e003229.full.pdf

Bluestone, J., Johnson, P., Fullerton, J. et al. Effective inservice training design and delivery: evidence from an integrative literature review. Human Resources Health 11, 51 (2013)

https://doi.org/10.1186/1478-4491-1151

Factors That Support Training Transfer: A Brief Synopsis of the Transfer Research. Will Thalheimer, PhD. WorkLearning Research, Inc. January 2020

https://www.worklearning.com/wpcontent/uploads/2020/01/Transfer-ofTraining-Quick-Research-Review2020-v1a.pdf

O’Donovan J, O’Donovan C, Kuhn I, et al. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open 2018;8:e021467. doi:10.1136/bmjopen-2017-021467

https://bmjopen.bmj.com/content/8/4/e 021467

The Current State of CHW Training Programs in SubSaharan Africa and South Asia: What We Know, What We Don’t Know, and What We Need to Do. One Million Community Health Workers Campaign & mPowering Frontline Health Workers. July 2014

http://1millionhealthworkers.org/files/2 013/01/1mCHW_mPowering_LitRevie w_Formatted.compressed.pdf

Rowe Alexander et al. Improving health worker performance: an ongoing challenge for meeting the sustainable development goals BMJ 2018; 362 :k2813

https://www.bmj.com/content/362/bmj. k2813

Kuule Y, Dobson AE, Woldeyohannes D, Zolfo M, Najjemba R, Edwin BMR, Haven N, Verdonck K, Owiti P and Wilkinson E (2017) Community Health Volunteers in Primary Healthcare in Rural Uganda: Factors Influencing

https://www.ncbi.nlm.nih.gov/pmc/artic les/PMC5372810/

88

Mott MacDonald | Mapping and Assessment

Classified as Internal

Performance. Front. Public Health 5:62.doi: 10.3389/fpubh.2017.00062 The effectiveness of mobile health (mHealth) technologies to train healthcare professionals in developing countries: a review of the literature. O’Donovan J, Bersin A, O’Donovan C. BMJ Innov 2015;1:33–36

https://innovations.bmj.com/content/b mjinnov/1/1/33.full.pdf

Mobile Technology in Support of Frontline Health Workers: A comprehensive overview of the landscape, knowledge gaps and future directions. Smisha Agarwal et al. John Hopkins University. 2016

https://www.researchgate.net/publicati on/305676655_Mobile_Technology_in _Support_of_Frontline_Health_Worke rs_A_comprehensive_overview_of_th e_landscape_knowledge_gaps_and_f uture_directions

Kononowicz AA, Woodham LA, Edelbring S, Stathakarou N, Davies D, Saxena N et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019;21(7):e14676

https://pubmed.ncbi.nlm.nih.gov/3126 7981/

Garcia-Elorrio E, Rowe SY, Teijeiro ME, Ciapponi A, Rowe AK (2019) The effectiveness of the quality improvement collaborative strategy in low- and middleincome countries: A systematic review and meta-analysis.

PLoS ONE 14(10): e0221919

https://doi.org/10.1371/journal.pone.0 221919

Improving postpartum care for mothers and newborns in Niger and Mali: a case study of an integrated maternal and newborn improvement programme

https://obgyn.onlinelibrary.wiley.com/d oi/full/10.1111/1471-0528.12818

M Boucar K Hill A Coly S Djibrina Z Saley K Sangare E Kamgang S Hiltebeitel USAID Applying Science to Strengthen Improve Systems University Research Co., LLC (USAID ASSIST‐URC) Horwood et al. A continuous quality improvement intervention to improve the effectiveness of community health workers providing care to mothers and children: a cluster randomised controlled trial in South Africa. Human Resources for Health (2017) 15:39. DOI 10.1186/s12960017-0210-7

https://human-resourceshealth.biomedcentral.com/articles/10.

1186/s12960-017-0210-7

Training and Supervision Did Not Meaningfully Improve Quality Of Care For Pregnant Women Or Sick Children In Sub-Saharan Africa | Health Affairs. Hannah H. Leslie Anna Gage Humphreys Nsona Lisa R. Hirschhorn Margaret E. Kruk: September 2016

https://www.healthaffairs.org/doi/10.13 77/hlthaff.2016.0261

Measuring the quality of supervisor–provider interactions in health care facilities in Zimbabwe. P Tavrow, YMI Kim, L Malianga - International Journal for Quality Health Care, 2002

https://pubmed.ncbi.nlm.nih.gov/1257 2788/

Henry JV, Winters N, Lakati A, et al. Enhancing the supervision of community health workers with whatsapp mobile messaging: qualitative findings from 2 lowresource settings in Kenya. Glob Health Sci Pract 2016;4:311–25

https://pubmed.ncbi.nlm.nih.gov/2735 3623/

89

Mott MacDonald | Mapping and Assessment

Classified as Internal

Ballard m, Montgomery P Systematic review of interventions for improving the performance of community health workers in low income and middle-income countries. BMJ, 2016

https://bmjopen.bmj.com/content/bmjo pen/7/10/e014216.full.pdf

Kok, M.C., Broerse, J.E.W., Theobald, S. et al.

Performance of community health workers: situating their intermediary position within complex adaptive health systems. Hum Resources Health 15, 59 (2017)

https://human-resourceshealth.biomedcentral.com/articles/10.

1186/s12960-017-0234-z

Serious Gaming and Gamification Education in Health Professions: Systematic Review. Sarah Victoria Gentry1 et al. J Med Internet Res 2019 | vol. 21 | iss. 3 | e12994

https://www.jmir.org/2019/3/e12994/

Mobile Technology in Support of Frontline Health Workers - A comprehensive overview of the landscape, knowledge gaps and future directions. Johns Hopkins University, 2016

https://www.researchgate.net/publicati on/305676655_Mobile_Technology_in _Support_of_Frontline_Health_Worke rs_A_comprehensive_overview_of_th e_landscape_knowledge_gaps_and_f uture_directions

Financing Global Common Goods for Health: When the World is a Country. Gavin Yamey et al. Health Systems and Reform 5(4):334-349, 2019

https://www.tandfonline.com/doi/full/10 .1080/23288604.2019.1663118

90

Mott MacDonald | Mapping and Assessment

Classified as Internal

Annex 3 - Scoring Criteria Through our literature review, web review and key informant interviews, we have identified some prominent examples of innovative solutions to frontline health provider learning and performance. We sought examples which used blended and innovative approaches, including digital, and which had shown signs of adaptation during the COVID pandemic. This is not intended to be an exhaustive list, but rather the beginnings of a database which can grow and evolve over time.

To support decision-making within Gavi, we have conducted a basic assessment of the interventions along three dimensions: effectiveness, scalability and sustainability. The scoring criteria are described below: Effectiveness – scored out of 5, according to categories

1 = No data – which might be because an intervention is undergoing an evaluation 2 = Programmatic data on enrolment – data on numbers completing, passing post-module quizzes 3 = Programmatic data on outputs – programmatic data on improved aspects of job performance 4 = Scientific Study: process – published scientific study with data on improved aspects of job performance, such as coverage 5 = Scientific Study: outcomes – published scientific study with data on improved quality of care or health outcomes

Scalability – scored out of 5, with points accumulated as follows:

Ease of technology employed (infrastructure & products): 2 = needing only basic mobile phone/ 2G; 1= needing smartphone / 3G/WIFI; 0 = high bandwidth required Geographic / language coverage: 1 = more than one country or language; 0 = one country/one language; Agility: 1 = modules or courses across several topics / COVID adaptation; 0 = intervention limited to one topic Scaling to date: 1 = 20,000 users or more; 0 = less than 20, 000 users or still at pilot stage

Sustainability – scored out of 5, with points accumulated as follows

Formalised government partnership 1 = Yes

0 = No

Government co-production & co-management / institutionalised learning 2 = Yes

0 = No

Government co-financing 2 = Yes

0 = No

91

Mott MacDonald | Mapping and Assessment

Classified as Internal

Annex 4 – Separate Document

92

Mott MacDonald | Mapping and Assessment

Classified as Internal

Annex 5 - Global Platforms There are a number of software platforms that can be used to create specific and bespoke digital tools for interventions. As these platforms become more widely used and invested in, they themselves evolve into something more powerful - with more functionality and expertise in how to enable the development of effective tools. Critically, they are also largely interoperable with each other, as well as with national systems such as DHIS2 – so the more they are used the more effective they become together. Whilst all are free and open source, there can be some ongoing maintenance / management / server costs. In fact, in order to ensure they continue to evolve and maximise their potential (make continuous incremental improvements and maintain data protection etc) this is something that is necessary in most instances.

Commcare, Dimagi Dimagi’s primary product, CommCare (www.commcarehq.org), is a user-configurable, open source platform that enables anyone to build mobile applications to support data collection, counselling, behaviour change, and a variety of other functions. It is the most widely-used offline data collection and service delivery platform. It has multi-platform functionality and can be used via mobile, web, and messaging platforms with bi-directional communication analytics which integrate directly with analytics tools for monitoring. It is interoperable with many of the other platforms detailed here.

With an average cost $2 per user per month, one study found the use of CommCare to be more cost-effective than certain vaccines, including the cholera vaccine. ReMiND (India) incurs an incremental cost of USD $205 per DALY averted and USD $5,865 per death averted.

It terms of reach and sustainability it has been deployed in some form in over 80 countries, nine of which that have made national scale commitments: Guatemala, Senegal, Burkina Faso, Benin, Ethiopia, India, Malawi, Mozambique, South Africa. Partners/clients include Unicef, BMGF, USAID, CRS, IRC, Partners in health, Oikoi, Abt, DSME, One Acre Fund, Medic Mobile, Rockefeller Foundation.

Community Health Toolkit, Medic Mobile The toolkit is open source platform which enables apps to be developed within it based on an existing blueprint. The products/apps serve as a public good for global health. Apps which can be developed in any language and the software supports health workers delivering equitable care that reaches everyone. It is interoperable with Moodle which can be used as back end with offline use and embedded gaming features.

Examples of its widespread implementation have helped health workers ensure safe deliveries, track outbreaks faster, treat various illnesses door-to-door, keep stock of essential medicines, and communicate about emergencies. Functionality includes messaging, digital decision support, data collection, task and schedule management, patient files and analytics. Health worker performance management functions allows users to take action based on real-time health worker performance indicators. It also allows health workers to view their own performance and compare to goals, peers, and previous time periods.

In terms of interoperability and sustainability, Medic Mobile's approach involves looking with a health system lens to support basic building blocks and aligning with national strategies to create more impact compared to just standalone tech projects.

93

Mott MacDonald | Mapping and Assessment

Classified as Internal

In terms of reach it has been very successful, with 33,000 HCWs using it in 15 countries, resulting in "over 1 million caring activities supported per month". Partners include University of Washington, d.tree, Harvard Medical School, I-TECH, Last Mile Health, Living Goods, Oppia Mobile and others.

OpenSRP Developed by a consortium of technology, implementation and academic partners, led by WHO’s Department of Reproductive Health and Research. An open source mobile health platform that allows frontline healthcare workers to electronically register and track the health of their entire client population. Used for health programme areas such as HIV, TB, Malaria, Reproductive, Maternal, Child and Adolescent Health, HPV and Child Immunization. It simultaneously provides programme managers and policymakers with current data for decision-making. The COVID-19 track and trace application is currently being tested and deployed in Indonesia with 200 healthcare workers.

It has won awards and UNICEF, Gates and Gavi have all invested in it. Gavi’s initiative on Innovation for Uptake, Scale and Equity in Immunisation (INFUSE) recognized OpenSRP as one of the “pacesetter” innovations of 2018.

In 2018, the Digital Square Global Goods mechanism selected OpenSRP as a mature software “global good.” Managed by PATH and supported by Gates, among other funding partners, the Digital Square Global Goods mechanism invests in scalable and interoperable digital health solutions that add value to countries’ health systems.

It complements and adds value to other “global good” digital health information systems, including medical records systems (OpenMRS), health management information systems (DHIS2), logistics information systems (OpenLMIS), and messaging platforms (RapidPro) that are often deployed at scale – and some of which are detailed here. It has Bluetooth data transfer for low connectivity environments.

It has been used across a variety of countries including Bangladesh, Pakistan, Indonesia, Tanzania and Zambia. Partners include mPower, Johns Hopkins University, Ona, Harvard University School of Public Health, Interactive Research and Development, Summit Institute, and Interactive Health Solutions under the THRIVE research study.

RapidPRO Developed by UNICEF, building on some of the issues they had with RapidSMS in operating in difficult environments, and allows the user to easily design, pilot, and scale services that connect directly with a mobile phone user without the help of a software developer. It collects data via short message service (SMS) and other communication channels to enable real-time data collection and mass-communication with target end-users, including beneficiaries and frontline workers. RapidPro also powers U-Report, UNICEF’s youth, and citizen engagement platform.

It is now used in 36 countries for real-time monitoring. It has significant reach as it integrates interoperably with widely used social media messaging platforms, including Facebook Messenger, Telegram, WhatsApp and Viber, as well as with DHIS2 and OpenMRS. Many of the digital health interventions powered by RapidPro have been linked into foundational digital health systems such as the DHIS2 and OpenMRS, resulting in easy user experiences for governments and reducing duplication and costs.

In Pakistan in 2018, families of 37 million children in 163 districts across Pakistan were reached through real-time information to increase demand for routine immunization with an initial focus in poorly performing polio tier 1 districts. Other initiatives have helped to coordinate cash assistance programmes; monitor water, sanitation, hygiene and nutrition interventions; and provide early detection and response to children with disabilities.

94

Mott MacDonald | Mapping and Assessment

Classified as Internal

DHIS2 Tracker Managed by the Health Information Systems Program (HISP) at the University of Oslo, this is an extension of DHIS2. Whereas DHIS2 is a HMIS system used for managing aggregated data, DHIS2 Tracker is used for individual patient level data at health facility/ community level. They now have COVID specific applications that are used for COVID-19 Case Management including screening and testing, contact registration and follow-up, port of entry screening and follow-up and as a surveillance system.

It is interoperable with DHIS2 as well as Electronic Medical Record (EMR) systems to allow for the sharing of clinical health data across facilities. It works via a mobile app called DHIS2 Capture. It is flexible and can be introduced as a pilot scheme initially, before being scaled up. Scaling up leads to increased operational costs in terms of support, devices and connectivity.

In Ghana, introduction of the DHIS2 Tracker has reduced health care worker workload by reducing reliance on paper based systems ensuring complete and timely reporting from the health facilities. Furthermore, Palestine's Maternal and Child Health (MCH) eRegistry has been used in more than 50,000 maternal health visits across 220 clinics in the West Bank and Gaza.

The success of this eRegistry has led the Ministry of Health to adopt DHIS2 for routine reporting and disease surveillance as well.

It has now been deployed in 52 countries with Covid-19 specific apps deployed in 32 of them.

Open Data Kit (ODK) Managed by Narfundi, ODK is a free open source suite of tools that allows data collection using Android mobile devices and data submission to an online server. It has been used for event-based Surveillance System and Healthcare Worker Training and Monitoring. ODK build is a drag and drop web-based form designer, best used for designing simple forms.

It is excellent for difficult operating environments as it can be used without an internet connection or mobile carrier service at the time of data collection. In Somalia, contact tracing data from 4,000 healthcare workers trained by WHO flows through ODK. It has been deployed in about a dozen countries and has 400,000 monthly users.

95

Mott MacDonald | Mapping and Assessment

Classified as Internal

mottmac.com 96