RELIEF WATCH Prototyping Trip Documentation Iraq, July 13 â€“ 19, 2019
Co-creating the Relief Watch â€˜service promiseâ€™ with a Sheikh in West Mosul
Table of Contents Executive Summary
Detailed learnings - Relief Watch Hotline
- Facebook Chatbot
- Feedback Forums
- Point of Service iPads
Summary of AAP Co-Creation Session
Executive Summary Over the course of one week, the Relief Watch project team tested five different channels for gathering Quality of Service feedback from affected people in camps and urban areas in Northern Iraq. The team also held a co-creation event with the Kurdish Relief Association, a local NGO with 14 years of experience, to develop new ways to accompany community feedback into action on-the-ground.
inefficient, inadequate, ineffective and/or inflexible. Through the various channels tested, the team collected a wide range of qualitative feedback from people affected by crises in Northern Iraq. The feedback was extremely broad in content, with issues such as a lack of basic food supplies being raised alongside complaints of inadequate medical facilities, and unqualified teachers in schools.
The disparity between organisations’ own perception of their work, and that of the communities in which they work, was striking. Everybody we spoke to had recent, first-hand experience of humanitarian assistance that was
The qualitative feedback gathered in each of the channels was presented to both Camp Management as well as the Head of Office of 4
“ Nobody comes here and sees our situation. This was the first time anybody asked us about these issues. It’s a kind of psychological support to us, you have heard our opinions.” - Participant, Women’s Feedback Forum, Salamiyah Camp, Ninewa, Iraq
OCHA in Iraq. The feedback was received with great interest from the HCT, who explained that somewhere within the system perspectives like this are being lost. The channels tested with communities to gather their input were group forums, hotlines, Facebook chatbots, and point of service iPads. Generally, the quality of the feedback diminished with the more technological channels. However, they were warmly received by group forum participants
who often followed-up with additional feedback in the days after the initial feedback forums Through conversations with humanitarians and affected people in Iraq, the team gained further insight into the ideal governance of Relief Watch, as well as the architecture of the website. These include:
- Testing the need for Relief Watch to be interfaced with, or linked to, the formal humanitarian system, is essential. The hypothesis that making community feedback publicly accessible, shortens the distance between donors and communities, and thus creates greater incentives for organisations to listen, needs to be tested. If no follow-up to feedback is seen, people will lose trust in Relief Watch, and user retention will drop.
- To be truly user-centred, a pilot should be limited in geography, but not in thematic scope. Allowing recipients of aid to set the parameters of the feedback shapes the conversations at the strategic level of the HCT. - District-level insights is the level of detail required to ensure responsible partners are held to account, and insights are actionable. Satisfaction rates for larger geographies are less useful for humanitarian decision-making.
- A cluster contact database is already curated by OCHA in Iraq. It could be used to ensure service insights are sent automatically to responsible partners. Interfacing Relief Watch with sub-national ICCGs, or bilaterally with organisations may also ensure a follow-up to feedback is seen.
- Getting an accurate portrayal of the situation on-the-ground, from the perspective of recipients of aid, is very difficult for humanitarian decision-makers to attain. This kind of information can guide their daily work 6
if it gives both a macro overview, as well as granular, qualitative details of what the situation on the ground is actually like. - A combination of group forums with digital touchpoints such as Facebook and WhatsApp will allow Relief Watch to be gradually scaled. This document outlines in further detail the learnings from each of the prototypes tested, as well the priorities for continued development of the Relief Watch accountability service. More detailed learnings can be found in the Key Learnings section on page 20, which is followed by a short outline of the next steps for the project before returning to Iraq in October.
We had community meetings here before, but nothing ever changed, so we stopped holding them” Resident, Khazer Camp
Testing a hotline feedback service in collaboration with UNOPS’ IDP Call Centre in Khazer Camp, Iraq.
“It’s very easy! If it helps, I would use it” Resident, Salamiya Camp
A family in Salamiya camp use a Facebook chatbot to provide their feedback conversationally
Prototyping Objectives Gaining an evidential understanding of the channels affected people would use to provide their feedback, and limitations and advantages of each channel, was the primary objective of this trip. The team wanted to explore what effects a ‘human filter’ would have on community feedback provided through offline channels such as group forums. Direct input channels such as Facebook chatbots and point of service iPads were also tested to discover the quality and utility of the feedback these mediums would provide.
humanitarian system, cluster meetings, bilaterally with organisations, with the HCT, or elsewhere, to ensure organisations become more responsive, and trust in the Relief Watch service is maintained. The focus of the trip was to learn experientially, through testing prototypes of various touchpoints, and providing a space for people to give their feedback on the channels used.
Another key objective was to explore how and where the service might interact with the 9
I don't expect any answer back from them, nothing will change. The best response would be seeing that our voices made a difference" Resident, Baharka Camp
A mother in Baharka camp gives her feedback on services in her area over the phone to a Relief Watch hotline operator.
Relief Watch Hotline
In collaboration with UNOPS’ IIC Hotline, the team developed a script that allowed callers to give their feedback on services in their area. Callers could steer the direction of the conversation towards any service they wanted, in order that they set the priority of the feedback. Operators asked callers three key questions:
- Rating from 1 –5 is not intuitive for callers. - Operators need to be better-trained in empathetic and active listening. - Interactive Voice Recordings (IVR) at the beginning of calls are not intuitive and a huge barrier for callers.
- What do you want to tell us about health* services in your area?
- On-boarding materials need to de-risk the call for people. “What will happen when I call?”
- Overall, how would you rate your satisfaction with health* services in your area?
- The hotline should allow callers to remain anonymous.
- Do you have any suggestions for the health* services in your area? 11
* education, water, cash etc., as decided by the caller
Chatbots respond automatically to users’ conversational input. They have been used across many sectors, including the humanitarian sector for many years. Traditionally, they have been deployed over SMS, but in recent years, with new updates to Facebook Messenger, new possibilities to engage with communities have emerged. Using an open-source library from UNICEF called RapidPro, a small Facebook chatbot was created. It allowed users to provide their rating and qualitative feedback on services in their area using Facebook Messenger Quick Responses in Arabic, Kurdish or English. Try the bot for yourself: m.me/ReliefWatch-Iraq 12
- Most people only write a few short words when given the chance to ‘write a review’. - Some people instinctively used voice-to-text functionalities to give feedback. This worked very well. - The idea of giving a review ( مراجعة, ), is not a commonly understood concept in Iraq. ‘Tell us your experience of…’ is a lot more intuitive. - Asking for a rating, conversationally worked very well, especially with ⭐ emojis. Instead of asking for a suggestion or review, users found it easier to explain their rating.
People in West Mosul and Khazer Camp using the Relief Watch Facebook Chatbot to provide feedback conversationally
Organisations just give and go. There are so many useless projects here. You provided the space to discuss this today” Resident, West Mosul
Left: Men’s Feedback Forum at Wasel Tased Community Centre, West Mosul | Right: Women’s Feedback Forum, Salamiyah Camp.
The team conducted two feedback forums with groups of 15 – 20 people in a community centre in West Mosul, and at a women’s support centre in Salamiya Camp. In both forums, participants were asked to set the agenda of the meeting by creating small discussion groups around services such as Health, Education and Water, as well as issues proposed by the group including Cash Assistance, Shelter, and Jobs. After discussing issues in small groups for a few minutes, everybody would have the chance to relay their thoughts to the facilitators. After two such rounds, participants were asked for feedback on the format, and how it could be improved. 15
- Participants in both groups remarked how therapeutic the experience was for them. It was the first time any of them felt heard. - Dividing into small groups wasn’t intuitive. The feedback was a mix of individual experiences as well as community-wide issues. - The quotes from these forums were the richest feedback the team collected. When humanitarians read the quotes, with the first-person voice preserved, they had an authenticity and immediacy to them that was very powerful.
Point of Service iPads
Gathering feedback at the point of service is generally considered the best method of understanding the quality of a given service. Many different sectors use PoS surveys to understand the quality of their service, from airport security, to Skype, and hospitals. The barrier to respond is very low, and the question can be tailored to the exact point of service. In a little over two hours, 68 people gave their feedback on their experience of the medical services in Baharka Camp, and 10 people wrote or voice recorded a short comment on their experience that day.
- Itâ€™s very hard to find a suitable location for the iPads, or even the UNHCR / ACTED hardware, at point of service in camps. The best method would be to have somebody with the iPad approach users as they leave the service point. This quickly becomeslabour intensive and expensive at scale. - Nobody gave feedback without being specifically asked to do so by the pharmacist or other NGO staff present. Similar UNHCR / ACTED hardware has been in place at various locations throughout the camp, and had received zero responses in over a year.
Left: PoS iPad in a Pharmacy in Baharka Camp | Right: ACTED / UNHCR PoS tool in Salamiya Camp
After testing different mechanisms of acquiring community feedback, the team hosted a cocreation workshop at a ‘makerspace’ in Erbil. Representatives from local NGOs, INGOs and UN agencies were invited to attend to give feedback on the concept, and brainstorm ideas on how and where Relief Watch could interface with the humanitarian system. The KRA Director and GBV cluster lead for Kirkuk who were present, welcomed the idea warmly. Through a series of exercises, the group created a concept that made local NGOs the Relief Watch ‘Focal Point’ for verifying the gaps in services elevated by the community. 18
- Local organisations don't attend cluster meetings because it is not relevant for them. Meetings are in English, and are very removed from on-the-ground realities. - Out of approximately 5000 local NGOs, only a few hundred are working with INGOs or UN agencies, partly because many of them are politically affiliated. Enabling trusted NGOs to elevate service gaps through the system could help improve service quality. - "International organisations have a monopoly on problems. They don’t allow voices of communities to go forward, so that they can get the money for their projects.”
“ One challenge I face as cluster lead for GBV issues is that all the experts
and international organisations come, but we don't find local authorities or communities who will participate in the meeting. This means that the gaps and information from communities don't make it to the meetings” Ayoub Alberber – GBV Sub-Cluster Lead, Kirkuk, Iraq.
Co-creation workshop at Erbil Makerspace with local NGO, Kurdish Relief Association.
Key Learnings - Analog channels can be scaled with digital touchpoints such as WhatsApp/Facebook.â€¨ Group forums were easily the most effective way of gathering feedback. After the sessions, people asked how they could follow up. This was then an opportunity for them to be onboarded to the Facebook chatbot, which often received messages in the subsequent days. Empowering communities to follow-up on group forums, with digital channels will enable the service to scale. As changes are seen on-the-ground, and trust is built with the service, communities can transition to monitoring organisations using Facebook Chatbots or
WhatsApp channels. Point of Service feedback inputs may become more relevant in Iraq over time, however Relief Watch should not rely on them for community input at first. - Feedback discussions should be organised by activity, not by organisation or cluster.â€¨ Prompting feedback discussions through an activity-based (or service-based) organising structure is both user-centred and helps bridge across the cluster-system. (e.g. Camp Management, Socio-Economic Support, Livelihoods, Reconstruction, Services (Health, WASH, Education). 20
A man in Khazer Camp gives his thoughts on the quality of his shelter, to a Relief Watch operator on the phone. 21
- Keep action traceable to the 'lowest level' possible. The Service Excellence Coordinator role is best placed to write a â€˜report cardâ€™ for organisations in their district. There is no need for an entirely separate auditor / watchdog. It will be important to keep Relief Watch small and contained at first. Report Cards could rate or review to what extent organisations are responding to the feedback / issues raised through Relief Watch, and how well they are trying to solve problems at the lowest level possible. - Once feedback goes up, it never comes down. As individual cases are escalated the likelihood of a positive outcome, and closing the loop with the original complainant diminishes.
- Integrity of user voices are rarely preserved up the system chain. Getting first-hand quotes from affected people to the upperlevels of the HCT is highly valuable. Quotes are used to steer discussions and guiding decision-making. - The missing link is action. The hypothesis that organisations will be incentivised to act when community feedback is publicly accessible to donors, needs to be tested. If communities do not see organisations responding to at least part of their feedback, trust in the service will be eroded very quickly. If action is not taken, interfacing Relief Watch with sub-national ICCGs, and bi-laterally with organisations might be a good option to continue exploring.
- Create space for communities to set their own feedback agenda. Giving people the space to set the agenda of town halls can help organisations understand community priorities, and uncover gaps in assistance. If communities continually raise jobs, cash assistance, or reconstruction work as the issues to discuss, this should steer the work of OCHA, and other organisations working with the Iraqi Government. This can also help some organisations answer the question, where should I be?â€¨ â€¨ Engage meaningfully with responsible partners. Making things actionable also requires visualising and connecting meaningfully with the responsible partners. There is a database of contact details for
cluster-leads etc that UNOPS are trying to integrate into the IIC. This could be one mechanism for getting issues to the right person. (e.g. 'Weekly insights from your district') - Refer the issues that need to be referred, not the issues the system asks for. A pilot of the service should be narrow in geography, but not have any limits on themes for feedback. The community to a large degree, dictate the scope / priorities. You can't limit the feedback of people, if they want to talk about illegal internments, the service should find ways to escalate those issues.
Maryam, a Relief Watch Service Excellence Coordinator, facilitates a Feedback Forum in Mosul. 24
Next Steps - Iterate the Service Excellence Coordinator role. Integrate responsibilities previously held by Watchdog / Data Gatherer role. Recruit consultants from previous trips to act in this role for one week in October.
- Create MVP website for â€˜mini-pilotâ€™ of the service in October. Plan for ~3 days of group forums, and ~2 days following-up with respondents through digital channels. All feedback should be live on website for organisations (and potentially the public) to access.
- Prepare workshop for actors from ICCG / NCCI / Sub-National Clusters and other organisations in Iraq to co-create the Service Excellence Coordinator role and further test Relief Watch portal interface.
- Create a video prototype of the service whilst in Iraq. The video should communicate how Relief Watch works from the perspective of an end-user and what impact and value it brings to recipients of aid as well as humanitarians.
Khazer Camp, Iraq
The Relief Watch team spent 1 week prototyping new ways of gathering community perceptions on humanitarian services in Iraq. This document o...
Published on Jul 25, 2019
The Relief Watch team spent 1 week prototyping new ways of gathering community perceptions on humanitarian services in Iraq. This document o...