mHealth Matters: People, Money & Performance by Erin Schiavone

mHEALTH MATTERS: PEOPLE, MONEY & PERFORMANCE CASE STUDIES FROM AFRICA

Recommended Citation: The PMI Africa Indoor Residual Spraying Project. October 2015. Kolyada, Lena, Elana Fiekowsky, Beth Brennan, and Keith Mangam. mHealth Matters: People, Money & Performance Case Studies from Africa. Bethesda, MD, Abt Associates Inc. Contract: GHN-I-00-09-00013-00 Task Order: AID-OAA-TO-11-00035

Abt Associates Inc. 1 4550 Montgomery Avenue 1 Suite 800 North 1 Bethesda, Maryland 20814 1 T. 301.347.5000 1 F. 301.913.9061 1 www.abtassociates.com

PMI | AFRICA IRS (AIRS) PROJECT INDOOR RESIDUAL SPRAYING (IRS 2) TASK ORDER SIX

mHEALTH MATTERS: PEOPLE, MONEY & PERFORMANCE Case Studies from Africa

CONTENTS Acknowledgments ......................................................................................................................... v Introduction ..................................................................................................................................vii Case Studies................................................................................................................................... 1 Africa: Using Mobile Payments for IRS Workers ............................................................................................... 3 Angola: Using Smartphones for Spray Data Collection .................................................................................... 6 Benin and Nigeria: Job Aids and Task Reminders via SMS ............................................................................... 8 Mali: Efficacy of Mobile Messaging to Prepare for Spray Campaign .............................................................10 Senegal: Smartphone-based Supervision for Real-time Management ..........................................................13 Madagascar: SMS-based Performance Monitoring Tracking for Real-time Management ........................15 Way Forward ............................................................................................................................... 19

III

ACKNOWLEDGMENTS This publication was made possible through funding from the Presidentâ&#x20AC;&#x2122;s Malaria Initiative to the PMI Africa Indoor Residual Spraying Project. The authors gratefully acknowledge the role of colleagues from partner technology firm Dimagi, Inc., which contributed to the design and implementation of some of the case studies described in the paper. The authors also thank Dr. Nduka Iwuchukwu, AIRS Tanzania Chief of Party; Pamela Riley, Lead mHealth Advisor at Abt Associates; and Laura McCarty, Senior Communications Manager for the PMI AIRS project, for their thoughtful review and feedback that enhanced the quality of the report. Finally, the authors appreciate the time and effort that Linda Moll provided in editing and Maria Claudia De Valdenebro for bringing this publication into its final form.

INTRODUCTION The aim of this publication is to demonstrate the potential of mobile-based technology (mHealth) to improve the quality and efficiency of public health programs, in particular indoor residual spraying (IRS) programs. To do so, it describes mHealth innovations implemented by the Presidentâ&#x20AC;&#x2122;s Malaria Initiative Africa Indoor Residual Spraying (PMI AIRS) Project.

IRS as A Malaria Control Strategy IRS is the spraying of insecticide on the walls, ceilings, and other indoor resting places of mosquitoes that transmit malaria. Since 2008, with help from PMI, many African countries have used IRS as a key strategy to prevent malaria. The PMI AIRS Project, led by Abt Associates, manages spray operations and logistics in 11 countries where malaria is endemic, and provides enhanced entomological monitoring in four additional countries. Spray operations are highly complex, requiring meticulous planning at the national, district, and village levels. This entails conducting geographical reconnaissance to identify work sites, procuring insecticide and equipment, managing warehouses, and training large numbers of local staff to spray homes in accordance with environmental and health guidelines. Implementation of spray operations involves dispatching a massive amount of equipment and materials as well as workers to the villages and towns targeted for spraying, according to set schedules. Tight spray calendars are driven by the need to complete IRS prior to the rainy season because rains complicate the logistics of delivering IRS and increase refusals of beneficiaries, who must move household items outside while their homes are sprayed. Teams of spray operators may stay for a few days in the same remote area to work in the villages that are difficult to access on a daily basis if travel from the main IRS base is not feasible. To ensure IRS operations are on schedule, closely monitored, and implemented in line with standard operating procedures, the PMI AIRS Project introduced several mHealth tools. Spray data collection, Senegal. Photo credit: Elana Fiekowsky

VII

mHealth in IRS Mobile-based technologies can intervene in public health programs at three levels: the beneficiary/individual level, the service provider level, and the program management level. In addition, mobile money services provide an opportunity to improve efficiency and accountability of financial flows in health programs. The PMI AIRS Project had the opportunity to pioneer applications in all four areas. With each PMI AIRS country program employing from 500 to 3,700 seasonal workers, orchestrating efficient spray operations is critical to delivering high-quality, cost-effective, and on-time spray campaigns that reduce the transmission of malaria just prior to or at the onset of the rainy season. The use of new tools like mobile phone-based short message services (SMS) and supervision helps to do this.

IRS Beneficiary Communications The PMI AIRS Project attempted to capitalize on the penetration of mobile phones in Mali to obviate the high costs and difficulties of beneficiary mobilization, traditionally done through door-to-door visits. The AIRS Mali team piloted a mobile messaging (text and voice) mobilization approach in Koulikoro District, trying out alternative methods of mobilization to effectively prepare households for IRS, while reducing the number of mobilizers.

mHealth for IRS Service Providers mHealth provides a useful channel to enable operations staff to receive real-time information such as reminders and alerts through mobile phones. Through information sent from project managers to team leaders and then to spray operators, the Project piloted mass SMS to seasonal personnel working in Nigeria and Benin. The purpose was to remind them of performance and environmental compliance requirements and to encourage their continued good work. Managers also send specific SMS alerts and reminders to various cadres of workers to address particular issues.

mHealth for IRS Program Management At the management level, technology can be a great support in tracking the implementation, resources, and quality of IRS program interventions. Supervision is important for maintaining safe and high-quality spray performance. The PMI AIRS Project had a difficult time addressing quickly the findings from spray performance supervision when relying on paper-based forms alone. Supervisory staff often suffered â&#x20AC;&#x153;form fatigueâ&#x20AC;? from the multiple reporting forms they were expected to complete and submit. Furthermore, use of paper forms led to errors when recording issues, reporting the information on time, and addressing in a timely way the problems revealed by the data. To improve supervision, the Project built a mobile application that reports on IRS performance compliance throughout the spray campaign.

VIII

To monitor daily spray progress and coverage, the Project has also been relying on paper-based data collection and reporting. Project staff spent significant time transporting paper forms from remote villages to data centers and entering the data into an Excel spreadsheet. Alternatively, a PMI AIRS operations manager would call each site – that had phone service – to ask for their spray progress and then enter that data into a spreadsheet. This delayed operations manager’s access to the data and prevented the operations manager from ordering immediate corrective action to improve the ongoing campaign. In some cases, a spray team had left a village before it received such instructions. In 2014, the automated system, in which team leaders send their team’s daily output data to a cloud-based database, allowed country-level and U.S.-based management teams to analyze the data on spray progress and order immediate corrective action if needed.

Mobile Payments in IRS Spray personnel are also able to receive payment through their mobile phones, and use the phones to manage their financial accounts including money deposit, transfer, and cash out. IRS campaigns typically employ hundreds to thousands of seasonal workers including spray operators, warehouse managers, and district coordinators. Distributing payments to so many employees is difficult anywhere, but even more so in remote spray sites. A functional banking system is often absent in remote spray areas and, therefore, cannot be used for paying wages. Prior to mobile money transfers, the PMI AIRS Project had to hire a secure vehicle to transport project staff who carried bags of cash to many remote operational sites, a costly, timeconsuming, and risky way to deliver seasonal worker payments. Mobile payments solved these problems, offering a more cost-efficient, fast, and safe – as well as transparent – way to move funds. The case studies presented below demonstrate how the Project has used mHealth to enhance spray performance and results at the levels described above by improving communication, supervision, output monitoring, and financial transactions. Currently, most of the pilots are successfully being incorporated into the permanent spray campaigns in nine PMI AIRS country programs.

CASE STUDIES

AFRICA: USING MOBILE PAYMENTS FOR IRS WORKERS Background AIRS country programs employ up to 3,700 seasonal workers for a weeks-long spray campaign each year. It is challenging to organize bi-weekly payments for such a large number of field-based IRS workers. Spray locations are mostly rural, often dispersed in remote villages, where the banking system is not well established or totally absent. AIRS country programs needed cost-efficient, fast, and safe ways to deliver payments. Initially, in most countries, finance assistants carried large amounts of cash into the field to pay workers facing the risk of theft, fraudulent activities, or loss of significant project funds. Later on, in some countries, the programs wired funds to the microfinance institutions based in the spray districts to organize payments for a small fee. In other countries, the programs contracted with local banks to disperse the payments based on worker attendance lists. More recently, making mobile payments through telecom providers became the third - and the most efficient and transparent - option.

Implementation Since 2013, six AIRS country programs (Ghana, Madagascar, Mali, Senegal, Zambia, and Zimbabwe) have been using mobile money payments. In Ghana, the AIRS country team works with telecom provider MTN. The project established a contract with MTN, which included a total amount to be delivered to the IRS campaign workers. The provider charged a small (usually 2-3 percent of the total amount) transfer fee for the service. AIRS Ghana provided to MTN the mobile money account numbers of each seasonal worker along with the amount to be transferred and a transfer schedule. It also informed the telecom provider of each worker's location, so that MTN would ensure their agents had adequate cash at the time of the payment in case many recipients wanted to cash out some or all of their earnings. In Madagascar, the AIRS program paid seasonal workers through the mobile provider AIRTEL via electronic wallet and a SIM Screen of a mobile money account on a phone, Ghana. Photo credit: card, a risk-free and reliable payment system. If a Lena Kolyada seasonal worker did not have a mobile money account, the project members helped him or her to open one. Some AIRS country programs also began using the mobile payment method with small local vendors based in the spray districts. These vendors, usually cooks and suppliers of consumables such as milk, juice, bread, and small IRS supplies, are used to operating in cash and often do not have a bank account. By introducing them to mobile money, AIRS programs expedited the paperwork and payment processes required for local procurement and ensured on-time delivery of meals, other food products, and supplies during spray operations. 3

Results All AIRS country programs that tested mobile money to pay seasonal workers have continued using this method of financial disbursement. Specific benefits include the following: 

Established regular, on time payments to seasonal workers, which both motivated the workers and eliminated the risk of work stoppages when payments were delayed



Reduced administrative and logistics burden



Significantly reduced the security risk of sending staff into the field with large amounts of cash



Significantly reduced the cost of transacting the payment; for example, in 2013, AIRS Madagascar realized a 78 percent cost saving over 2012, when it used finance assistants to distribute cash payments



Increased transparency, thus minimizing opportunities for financial fraud.

AIRS team member is verifying with AIRTEL staff lists of payment recipients, Madagascar. Photo: Anna Maria Paddack.

Lessons Learned 

The main lesson learned is to consider expanding this model of payment to all aspects of the country program whenever appropriate. It is an easy, low-cost and user-friendly approach to execute payments if all components of the process are intact including availability of required funds with the mobile money agents and timely transfers to a bank.

Conclusions As a program management tool, use of mobile money greatly improved operational efficiency, lowered transaction costs and risks, and enhanced accountability of the program. In addition to improving program management, the introduction of mobile money to the IRS programs benefited telecom providers and individuals. With respect to providers, AIRS country programs helped the mobile network operators to expand the reach of their platforms and to increase agent locations by using their systems in the remote areas on a more regular basis.

For beneficiaries, the PMI AIRS Project helped expand the growing mobile money culture from urban to rural residents, thereby increasing opportunities for overall household economic wellbeing. The seasonal workers could use the electronic funds in various ways, such as paying school fees, buying airtime and taxi services, and as a safe storage mechanism, thus increasing household savings.1 Further, mobile money could empower certain household members, women in particular, who have traditionally had less bargaining power.2 By paying the increasing number of women IRS workers in mobile money, expansion of the mHealth application can increase their financial abilities and decision-making authority. For many women it eliminates a third party who controls the money, when accessing the money through the mobile phones. Direct access to the electronic funds will allow for more space for independent financial planning, saving and purchasing.

Jack, William and Tavneet Suri. January 2011. Mobile Money: The Economics of M-PESA. NBER Working Paper No. 16721. JEL No. O16,O33,O55 Available online at: http://www.nber.org/papers/w16721.pdf Accessed on July 13, 2015. 2 Ibid 1

ANGOLA: USING SMARTPHONES FOR SPRAY DATA COLLECTION Background Mobile applications can increase accuracy of reported data, provide access to real-time data, and eliminate the costs for paper-based data processing and storage centers, data entry clerks, and data transport in low-resource settings. For the 2013 IRS campaign, AIRS Angola piloted a mobile data collection and verification system as an alternative to the traditional, paper-based data collection and quality assurance tools in Bailundo Capital, a commune in Huambo province, Bailundo Municipality (approximately 15,000 targeted structures).

Implementation The AIRS pilot was designed to test the feasibility of mobile data collection and quality assurance for use in IRS campaigns. For the AIRS Angola IRS campaign in October 2013, it began by providing spray operators with a smartphone and their team leaders with Android tablets. Using a mobile-based platform AIRS Angola reduced data entry errors by preprogramming smartphones with electronic versions of the program’s standard IRS data collection tools. These forms were built with validation checks to prevent skipping mandatory fields and to reduce data errors commonly found on paper forms. Team leaders used tablets to immediately validate data submitted electronically in lieu of mass verification of the entire team’s paper forms at the end of each day’s spray activities. With this system, the AIRS Angola M&E and Operations teams, the chief of party, and the U.S.-based M&E specialist could access the data in real-time, identify issues, and recommend appropriate and speedy corrective actions. Initially, AIRS Angola tasked 48 spray operators to record data on smartphones from roughly 12,000 households and to electronically submit the data to the cloudbased system. Eight team leaders were given tablets to simultaneously supervise staff performance and verify data quality. Twentythree staff exhibiting poor understanding of the mobile system due to poor technology literacy were reassigned to paper-based data collection within four days of the pilot startup. Photo: Elana Fiekowsky

Results Once the spray operators confirmed that their spray data were uploaded to the server, the team leader opened the tablet and viewed the verification table that listed the key indicators for their spray teams for that day. Since the data were available immediately, the AIRS Angola M&E team was able to pull and/or clean the data for that day and estimate spray progress in the main office in Huambo. If the application identified a problem in the data, the team leader flagged the spray operator form electronically, noting the issue in a text comment for the M&E team’s attention and correction. For example, based on such notifications the M&E team noticed a higher than expected refusal rate in a few villages. Accordingly, AIRS Angola, in collaboration with the Bailundo Municipal Department of Health, deployed a mobilization unit the next day to circulate in those and surrounding villages. The unit used a loudspeaker to disseminate IRS messages in both Portuguese and Umbumdo, to increase awareness and willingness to participate, and IRS acceptance rates increased in these areas when spray teams returned to those villages to complete the spraying.

Lessons Learned 

Low-literacy workers need more training and hands-on supervision. The team recommends adding to the three-day training from two to three extra days for these workers.



The pilot occurred in a new IRS area, thus new spray operators were learning both spray skills and phone data entry skills. In the future, the Project recommends implementing mobile data collection with personnel experienced in IRS and allocating more time for training.



The spray operators submitted data for all structures sprayed in one day at the end of the day rather than after each structure, which made data validation difficult. In the future, the system should allow spray operators to submit data after every structure is sprayed so team leaders can validate data throughout the day rather than uploading all of the day’s data at once.



Technological “bugs” (i.e., filtering and sorting difficulties) in the team leader verification system during the first week of spray made data supervision difficult. The PMI AIRS Project recommends further development of a modified data aggregation system and testing it in advance of implementation in order to catch similar bugs.

Conclusion Despite some problems during early stages of implementation, the advantages of shorter delays between data entry and data analysis were clear. The mobile system successfully allowed the AIRS Angola team to detect several performance issues and adapt operations to address the problems by the next day. However, there was no cost analysis conducted to evaluate efficiency of this method as compared to the paper-based collection. The data verification has increased working hours for team leaders at the end of the day.

NIGERIA: JOB AIDS AND TASK REMINDERS VIA SMS Background One of the biggest challenges to implementing an IRS campaign is supervision, and supervision is impossible without good communication. During a spray campaign, PMI AIRS country programs train and hire on average 1,300 seasonal workers to do various jobs including spray homes, supervise spray teams, manage insecticide storage, collect data, and wash uniforms. Previously, the most expeditious way to communicate with the entire field staff was the phone tree. For example, if an operations manager noticed that spray operators were not correctly wearing protective clothing, he would call team leaders (up to 50) who would relay the message to their workers during the following morningâ&#x20AC;&#x2122;s meeting. It was not an efficient system.

Implementation In 2013, AIRS Nigeria hired and trained more than 300 seasonal staff for its IRS campaign throughout two large Local Government Areas. To tackle this communication and supervision challenge, it decided to use mass SMS to reach out to all team leaders and other categories of staff at once. The country program enrolled all spray personnel in a mass SMS platform and began using text messages to disseminate reminders, reinforce training, and motivate staff. In addition to all-staff messages, AIRS segmented workers by job title and geographical area, and sent them tailored messages to improve their performance. If project technical staff identified a problem, they could immediately have supervisors notify the affected staff to point out the problem and how to correct it. For example, when technical staff discovered that spray operators were not using the proper spraying technique and were applying too much insecticide, a mass SMS was sent to all operators to remind them of the correct behaviors. During follow-up visits, program supervisors confirmed that workers had adjusted their spray technique and insecticide application according to recommendations.

A female team leader uses SMS to communicate with the operations manager, Doma Local Government Area. Photo credit: Nduku Iwuchukwu

To set up the platform, the program used a local technology firm SMS Africa. It is a commercial platform, which is open to the public upon payment of a fixed rate for bulk messages. , . The program

also collected phone numbers of all hired workers and organized them by job and location. To run the SMS-based communication smoothly, the country programs had to purchase significant amount of airtime from the telecom provider. Over the course of its 33-day IRS campaign, AIRS Nigeria disseminated one or two messages per week. The messages were about recruiting, the campaign start-up, correcting problems observed in field operations, and general performance reminders. Toward the end of the campaign, when workers were becoming fatigued, the messages included words of encouragement and recognition to motivate workers to finish strong.

Results While no quantitative data were collected to document the impact of this mHealth innovation, workers reported that they found the corrections and reminders helpful and informative. Supervisors said the SMS program eased their supervisory work. “The text message program made supervision much easier. Once we identified a problem, we could immediately notify spray operators and make corrections. It has definitely improved our quality of work,” reported Nduka Iwuchukwu, AIRS Nigeria Operations Manager at the time of the SMS program implementation.

Lessons Learned 

Not all phone owners use the same telecom provider, and each telecom provider has different service plans and coverage areas. These need to be factored in when procuring airtime from a telecom provider.



Airtime needed was more extensive and expensive than expected. A country program needs to plan the number of SMS it expects to send out during the campaign, so that it budgets and purchases enough airtime for message transmission.



Open-source, low-cost, and more user-friendly platforms such as TextIt are recommended to replace commercially-provided systems.



Not all SMS go out at the planned time. During periods of high network traffic, the order of SMS may change, with messages sent later in a day transmitted before the earlier ones.

Conclusion The use of the SMS platform for communicating reminders and corrections, as well as motivational messages, to large cohorts of IRS workers promises to improve outcomes. The role of the country program leadership in ensuring effective resources for adoption and proper integration of this innovation remains critical to its success. In addition, the use of the SMS platform for reminders and other messages can be easily adapted for a broader audience of public health implementers.

MALI: EFFICACY OF MOBILE MESSAGING TO PREPARE FOR SPRAY CAMPAIGN Background In Mali, residents traditionally have learned about upcoming IRS campaigns through door-to-door mobilization, in which community field agents (mobilizers) are engaged and trained by the country program to deliver messages about the campaign. In 2013, over 1,000 mobilizers covered three districts. Careful supervision of such a large number of mobilizers over a dispersed area was challenging for the program’s operations manager and monitoring and evaluation (M&E) manager. As a result, mobilizer success in some areas was discovered to be far inferior to what the mobilizer themselves had reported. In addition, door-to-door mobilization is very expensive in terms of financial and human resources.

Implementation To reduce costs while still imparting information to households about what they needed to do both before and after spraying, AIRS Mali piloted a mobile messaging (text and voice) mobilization approach in Koulikoro District. The pilot tested alternative methods of mobilization that were intended to effectively and equally prepare households for IRS, but cost less than the traditional mobilization approach. Specifically, it assessed the feasibility of replacing door-to-door mobilization with mobile messaging. To measure the alternative approaches’ efficacy, the pilot study assessed household preparedness in three villages in Koulikoro District (Tienfala Village, Tienfala Gare, and Fougadougou) where mobile messaging was piloted against three comparison villages (Fassa, Wolongotomo Socura, and Wolongotomo Socoro) that were mobilized solely through a modified door-to-door approach. The modified approach incorporated radio outreach and town-hall style announcements in addition to traditional door-to-door communication. Households in all three pilot villages received SMS, in two of the villages, Tienfala Gare and Fougadougou, households also received voice messages. The AIRS Mali M&E team went to the three pilot villages to collect phone numbers from 673 residents in all of the 570 structures targeted for spraying. After collecting the phone numbers, a data entry clerk uploaded the numbers into the TextIt platform, a software application that allows a single user to send short SMS to a large number of mobile phones via a computer. Based on the locations of the individuals whose numbers had been collected, the team determined which language to use when sending messages to specific groups of beneficiaries as well as group numbers in specific villages, to effectively target messages. The AIRS Mali team sent four types of messages to beneficiary households before, during, and after the IRS campaign. “Mobilization” messages were sent 10 days before the campaign to remind beneficiaries of the benefits of IRS and to let them know the timing of the spray campaign in their village. “Alert” messages were sent three days before the start of the campaign to warn them of the dangers of malaria and how they can protect themselves from malaria, such as accepting IRS. “Instruction” messages were sent during the spray to tell residents how to prepare themselves and their structures for the spraying, as well as what to do immediately after spraying. Finally, “Advice” messages were sent a few days after spraying to inform beneficiaries about signs of insecticide poisoning and other actions to avoid that would interfere with the efficacy of the insecticide.

Results Before full implementation of the mobile messaging approach began, a mini-pilot was done to test the SMS and gauge beneficiary perceptions. This exercise showed that the addition of voice messages would be useful to the overall mobilization approach. Thus, in two of the three pilot villages, voice messages were sent in conjunction with the SMS. At the end of the full pilot, there were a total of 9,402 SMS sent out across three villages, and a total of 4,474 voice messages sent out across two villages. Both mobile pilot and comparison villages had high spray coverage (defined as percentage of found structures that were sprayed). The initial spray coverage in mobile pilot villages was 86 percent compared to 96 percent in the villages where traditional door-to-door mobilization was done. Similarly, structure preparation was lower in villages mobilized via the mobile messaging (47 percent vs. 78 percent). The most common reason for structures not being sprayed in mobile messaging mobilization areas was householdâ&#x20AC;&#x2122;s â&#x20AC;&#x153;refusal,â&#x20AC;? not necessarily because they opposed IRS but because they did not want to miss a day in their farm field or spend time removing all of their belongings from the structure. For beneficiaries who were hesitant about having their structures sprayed, it was much easier to ignore the mobile message than a door-to-door mobilizer on the day of spray. In person, the mobilizer can work directly with the beneficiary or village leader to gain acceptance. In this pilot, the mobile messaging mobilization approach was both more costly and less A Spray Operator is entering IRS data in a smartphone, Senegal. Photo credit: Elana Fiekowsky efficient than the door-to-door approach. As the table shows, the average percentage of structures prepared upon first visit by spray teams in the mobile messaging-mobilized villages was 47 percent compared to 78 percent in villages mobilized door-to-door. The average cost incurred per structure prepared (meaning the structure had been mobilized and was ready to receive the IRS team on the day of spray) was $6.55 per structure in the mobile messaging villages compared to $1.08/structure in the villages mobilized door-to-door.

Lessons Learned 

Mobile messaging mobilization for IRS had limited effectiveness because much of the pilot population was unable to read and understand the messages (although some illiterate residents could ask other residents to read the SMS message or explain voice message to them and thus they still got the information about IRS timing).



Mobile messaging is more effective if used in combination with a mobilizer or some other person who can answer questions about the spray campaign. Numerous beneficiaries wanted more information after receiving the messages, but there was no number to call or a point of contact to ask. The AIRS Mali team could partner with the National Malaria Control Program to set up a tollfree number that would provide more information.



The enumeration of phone numbers was very costly. In the future, it would be more cost efficient to work phone number collection into another activity already occurring in these villages, to share costs across activities.



Future attempts at mobile messaging should investigate partnering with locally based telecom providers since this pilot partnered with a US-based provider. By working with local partners, there is the potential to reduce the cost of sending mass messages.

Conclusion This pilot study was useful in that it provided an example of an mHealth approach that was not appropriate given contextual constraints. As noted above, the low literacy and strong dependence on human interaction and social ties in Mali do not provide an atmosphere receptive to impersonal mobile messaging. Traditional IRS mobilization in Mali leveraged this preference for social interaction; by eliminating the human touch, the overall success of the intervention suffered, with lower rates of acceptance and structure preparedness. While the results of this pilot do not encourage doing mobilization solely through text or voice messages for mobilization in Mali (or similar settings), the pilot is a stepping stone toward other, more effective implementation designs. For instance, mobile messaging might be complemented by a mobilizer traveling with an IRS team on the day of spray. The pilot also highlights the importance of performing a contextual analysis before choosing to implement an mHealth intervention.

SENEGAL: SMARTPHONE-BASED SUPERVISION FOR REAL-TIME MANAGEMENT Background Supervision of IRS campaigns is vitally important for maintaining safe and effective spray performance. In Senegal, team leaders, who directly manage the spray operators, are overseen by supervisors from the government’s National Hygiene Service and AIRS Senegal site supervisors. These supervisors used to collect data on paper forms – on topics such as spray preparations, spray technique observations, and stockroom performance – to manage environmental compliance and spray operations throughout the campaign. The forms were submitted to the AIRS country team for project monitoring and improvements. Using paper forms had several problems: errors in noting down the data on the report forms and challenges to promptly submitting the forms meant the country team could not process and share information in real time, nor could they efficiently identify and correct systemic shortcomings in spray operations. Consequently, improving spray behaviors and procedures was ad hoc.

Implementation To standardize project supervision across the project staff and government partners, the PMI AIRS Project adapted the paper supervisory forms into a mobile application. At the end of each spray day, government and project supervisors in the field use the application to complete and submit their forms to a cloud-based open-source platform. The AIRS Senegal database manager compiles the data and emails a comprehensive daily report to both the country- and international-level stakeholders. The reports are used to spot and address systemic deficiencies or “red flag” areas immediately. Work started in April 2014 to build, test, and deploy a mobile system for the July 2014 IRS campaign that would allow the Senegal and U.S.-based teams to digitize the spray supervision forms to ensure standardized supervision and immediate analysis. Dimagi, the PMI AIRS Project partner, built the system using its CommCare platform, an open-source mobile application that enables non-technical users to access Dimagi’s cloud platform and build their own mobile systems. Dimagi staff worked with members of AIRS Senegal’s Operations, M&E, Data Management, and IT teams to design a prototype CommCare Mobile Supervisory Application, test it, and refine it. The Mobile Supervisory Application was finalized after multiple rounds of feedback from the AIRS Senegal staff, the Dimagi team, the AIRS home office staff, the IRS Steering Committee on project prototypes, and stakeholders. Dimagi also held a training of trainers for selected AIRS Senegal staff. The IT and M&E team received more specific training to learn how to access daily reports from the mobile system. AIRS staff then led trainings for supervisors and district coordinators on how to use the mobile application. Dimagi staff also developed all of the necessary training guides and other materials so that the Senegal team could run their own campaigns in the future, with only limited remote support from Dimagi. Different guides were developed for 1) team leaders (primary users), 2) district coordinators, and 3) AIRS Senegal staff.

Results The mobile application enabled in-country and U.S.-based program senior management to better manage spray operations. The AIRS Senegal team recognized the importance of the automated and standardized supervision forms, and was making real-time corrections in the field based on incoming data. For example, the AIRS Senegal team was able expedite the morning mobilization and transport the spray operators to the field sooner. Thus, they had more time for spraying in the villages. Specific results for the CommCare Mobile Supervisory Application include: 

Standardized supervision protocols and quality controls for all campaign supervision efforts.



Ownership by government and AIRS Senegal supervisory staff of the use of supervisory tools.



Empowered government supervisors who use the automated, user-friendly job aid to do their jobs effectively, such as immediately sending all spray teams a reminder on setting correct spray pump pressure level after observing such an error repeatedly among spray operators in one team.



Improved campaign management from site level to the central level, as a result of immediate actions to address deficiencies found with the supervisory tool from the real-time field data.

Lessons Learned 

Allow adequate testing time for primary users to transition from the use of paper forms to mobile forms (i.e., changed question order, new skip patterns, and the way the results are reported in the mobile form)



Design a system for deploying the supervisory application updates during the spray campaign, while phones are in use



Increase the training and practice time for supervisors from one to at least two full days of skill building



Automate the reports in the future to ease the use of the system so the AIRS Senegal team will not need to pull the data, and enter the data into the tracker before sending it to the applicable stakeholders every day

Conclusion By automating data reporting, the mobile system gives field staff and AIRS management immediate access to data on spray campaign quality, which leads to a better-run spray campaign. Given the success of the Senegal pilot, the PMI AIRS Project will scale up this work to all spray countries. It should be noted that the project did not evaluate cost efficiency of switching to smartphone-based supervisory inspections but it was observed that government counterparts were much more involved and actively supervised when doing the work with the smartphones.

MADAGASCAR: SMS-BASED PERFORMANCE MONITORING TRACKING FOR REAL-TIME MANAGEMENT Background In the past, the PMI AIRS Project tracked performance using a paper-based system to collect performance data at every operational site. One of the system’s tools was the Spray Performance Tracker Sheet, a daily performance report that is filled out by hand by a team or sector leader or site manager and displayed at each site. It is also used to report data upward. At the end of the spray day, the country program’s operations manager would call each site to ask about their spray performance and then enter that data into an Excel spreadsheet. For the operations manager, the process of collecting individual site data via phone was cumbersome and time consuming, and because s/he could not reach every site each day, did not provide a complete picture of the project’s spray status on a regular basis. Another tool that helped with tracking performance was the spray data form that spray operators fill in daily and a team leader is required to physically deliver it to a data entry site at the end of the day for uploading to the PMI AIRS database. In Madagascar, the Project sprays in some very remote areas reachable only by boat and/or long hike. This makes it impossible for spray teams to deliver the completed spray data forms to the data entry site with enough time for data clerks to enter the data and analyze the spray coverage before the teams have moved on to other areas. AIRS Madagascar decided that for the 2014 spray round, it needed a reliable way to collect the daily operational data so the country team could immediately analyze and improve spray coverage. In that year, AIRS Madagascar also sprayed a new region of the country. In the initial year of IRS, spray teams often encounter a high number of refusals because beneficiaries are not familiar with IRS and with being asked to empty their house prior to spraying. AIRS Madagascar needed a way to closely monitor spray coverage in case the team needed to use more mobilizers or work with local village chiefs to increase acceptance.

Implementation Tapping the skills of its mobile technology partner, Dimagi, Inc., the PMI AIRS Project piloted the Performance Monitoring Tracker (PMT), an automated system that enables AIRS Madagascar’s team leaders to send SMS with their team’s daily output data to a cloud-based database. With that, the Malagasy and U.S.-based project senior management teams track and analyze spray progress data within districts and for the country as a whole at the end of each day. The real-time data allow them to spot where specific spray teams are not reaching their objectives and to make programmatic decisions to ensure high-quality spray coverage. In August 2014, Dimagi and AIRS Madagascar modified an SMS system that AIRS Senegal had used for two spray campaigns earlier in the year. The modified SMS system digitized the Spray Performance Tracker Sheet into an SMS-based PMT tool. As in Senegal, the system uses Dimagi’s open-source SMS tool CommConnect. CommConnect enables non-technical users to access Dimagi’s cloud platform and build their own mobile systems for SMS campaigns. A paper Spray Performance Tracker Sheet is still filled out by hand and displayed at each site for the spray actors in the field.

Over the course of the spray campaign, Dimagi staff worked with members of AIRS Madagascar’s Operations, M&E, Data Management, and IT teams to modify and test the SMS system. After a few rounds of feedback from the AIRS Madagascar staff, the Dimagi team, and the AIRS home office staff, the SMS system was finalized. One day of the week-long IRS training for sector leaders and district coordinators was devoted to SMS deployment. Members of the IT and M&E teams attended their own specialized training to learn how to access daily reports from the mobile system. Dimagi staff also developed all of the necessary training guides and materials so that the Madagascar team could run their own SMS campaigns in the future, with limited remote support from Dimagi. Different training guides were developed for 1) sector leaders (primary users), 2) district coordinators, and 3) AIRS Madagascar staff. For the second deployment, in the central highlands, all trainings were led by AIRS Madagascar staff and the deployment itself required only remote support from Dimagi.

A Team Leader sends PMT SMS at the end of the day after validating the data. Photo credit: Elana Fiekowsky

Following the two SMS deployments, Dimagi staff worked regularly with AIRS Madagascar to answer questions and address issues.

Results 

Immediate, aggregated data were available to AIRS Madagascar decision makers and PMI.



The data were used to more carefully follow the performance of the sprayers and monitor their use of insecticide.



The SMS tool’s built-in logic checks decreased the number of errors in submitted data. When data that was submitted above a certain threshold or was not logical given the number of spray operators, the system would send a message stating there was an error and asking the supervisor to resend.



Daily campaign progress could be monitored by a larger number of implementers including project management, government counterparts and PMI..



It was clear that the Madagascar team saw the importance of the immediate access to operational data, could monitor spray progress of remote spray teams, and was making corrections in the field based on incoming data.

Lessons Learned 

PMT system training should take place shortly before the spray campaign starts. If there is a long gap between the training and campaign start-up, users may need refresher trainings.



Linking the SMS system to the AIRS database would allow assessment of the reliability of the data being submitted.



Setting up a system that will generate and automatically send a daily record of all SMS sent and received by email would greatly facilitate monitoring of users.



Spray teams, and supervisors in particular, would benefit from more training on interpreting the data.



Teams should be trained to handle technical difficulties with the phones.

Conclusion The PMT pilot was successful in Madagascar. Due in part to the data reporting system, the AIRS Madagascar team managed a difficult spray campaign that achieved 96 percent coverage of targeted structures. With the lessons learned, the PMI AIRS Project and Dimagi are implementing the PMT system in all spray countries.

WAY FORWARD Starting in 2015, the PMI AIRS Project has been rolling out three mobile systems in countries that are implementing IRS: (1) mobile supervisory forms to assist in the monitoring of environmental compliance and performance supervision, (2) SMS-based daily reporting of operational data, and (3) the mass communication of daily reminders and motivational messages. The systems use a range of phone types, from Android to the simpler SMS-enabled brick phones. Implementing the three mobile systems will enable the project to engage with personnel at all levels of operations and in more remote areas, where internet connectivity is not available. In addition, the PMI AIRS Project is working on increasing number of country programs to adopt mobile payment systems. Through the experiences described in this publication, PMI AIRS will be able to capitalize on lessons learned and ensure improved performance across all countries.