Source: November 2013 face-to-face baseline survey, February-May mVAM phone surveys. Cross-tabulation of the Food Consumption Score and reduced Coping Strategies Index.
Could you imagine a doctor conducting a medical check-up without a stethoscope or a thermometer? As humanitarians, we attempt to track the needs of the communities we serve. Yet, data collection processes tend to be so expensive and time consuming that we are all too often, in effect, doctors attempting to treat an illness without up-to-date information on the patient.
In order to improve the quality of the data we collect on what our beneficiaries need, WFP is testing ways of implementing surveys remotely by contacting people through their mobile. Our experience, so far, shows that SMS can be an effective way to implement short food security surveys in eastern Democratic Republic of Congo (DRC). Thanks to a grant from the Humanitarian Innovation Fund, WFP is now implementing voice surveys in eastern DRC and central Somalia to collect food security data. In each country, our operators place calls every month to displaced people living in camps. The information is then used to help us gain a deeper understanding of how food needs are evolving.
What have we learned?
We chose to test voice surveys in DRC and Somalia, as these countries present two starkly different operating environments, and allow us to compare and contrast the identified lessons from the different settings.
In eastern DRC, we call survey participants who live in a camp where only one out of every four families has access to a mobile phone. Because mobile phones were not widely owned, we decided to distribute basic handsets to respondents. For many people, it was the first phone they had ever used. As there is no access to electricity in the DRC camp, we also set up a solar charging station where respondents could recharge their phones for free. By contrast, in central Somalia, where two-thirds of respondents already owned mobile phones, we have been able to place our calls without the additional assistance that was required in DRC.
After initial survey rounds, our general impression is that the approach works: we have been able to use phone calls to carry out a series of household surveys, in a complex humanitarian situation. We are able to produce information at higher frequency than before including, for central Somalia, an area of highly restricted humanitarian access.
These achievements appear to stem from the user friendliness of voice calls, even for the very vulnerable populations we are working with. In DRC, the average call duration has dropped from 10 to 7 minutes as respondents get more comfortable with our calls, and the initial indications are that respondents prefer short phone surveys to longer face-to-face interviews. In both countries, response rates have varied from 50% to 70%, in line with the results of other phone surveys panels in Africa and Latin America (World Bank, 2012). In both countries, we have obtained high response rates from women.
Response rates to phone surveys
Initiative / Response Rate (%)
Listening to Dar, Tanzania (25 survey rounds) / 66%-75%
Listening to South Sudan[C1] (4 rounds) / 52%-68%
Listening to LAC, Peru (Wave 1) / 48%-54%
mVAM, DRC, 5 survey rounds to date) / 45%-72%
mVAM, Somalia, 3 survey rounds to date) / 58%-63%
Source: World Bank, 2012; WFP 2014
The results of the monthly survey rounds we have carried out show that the data has the potential to support a better understanding of needs. Every month, we report on food security indicators such as the Food Consumption Score and the Coping Strategies Index – widely used proxies in the food security community. The survey we have for DRC immediately picked up a deterioration of food security indicators when food assistance was halted, and also captured typical seasonal variations well.
Of course, it has not all been smooth sailing: in DRC, we struggled somewhat to reach male survey respondents. We have faced technical obstacles: on some days, the phone network is saturated and our operators cannot get through. The charging station in the camp broke down on two occasions, requiring repairs and leading to downtime. And, of course, some of the phones we provided have been broken, lost or sold. Finally, our population of interest is mobile by definition, and some have left the camp to go back home since we started the survey – some of them to locations without phone service, others to places where they can still be reached. We are not yet sure whether this is an obstacle, or an opportunity.
What are we looking for over the next months?
The first months of the pilot have produced lessons about the process of using live calls for phone surveys. We now look ahead to amassing more evidence, including on broader impacts, as the pilot continues. We expect different stories to emerge from DRC and Somalia, providing important insights on the influence of context.
The introduction of interactive voice response (IVR) calls, which is imminent, will be a major milestone. IVR calls enable callers to interact through the use of voice recognition or keypad tones and provide responses with pre-recorded audio. They are cheap, scalable, and can be placed at any time of day, or during the weekend when operators are not at the office. We are curious to know whether certain demographic groups – such as the elderly – find IVR calls user-friendly. This will help us understand how to best use both live and automatic voice calls in our future data collection efforts.
A detailed analysis of the behaviour of our test panel will be necessary to understand what type of respondent has tended to drop out, and what factors can encourage people to continue participating.
As we are at the early stages of the project, we are not able to say whether remote mobile voice calls constitute the proverbial stethoscope around the doctor’s neck. While we have documented and disseminated results, including to decision-makers, more experience with this new data stream is needed before we can confidently say that phone surveys lead to better decisions and outcomes for the vulnerable communities we work with.
Jean-Martin Bauer, Marie Enlund, Silvia Passeri, Lucia Casarin
Check out this video about WFP's mVAM project.
Listening to Dar project
Croke, K. et al (2012) Collecting high frequency panel data in Africa using mobile phone interviews.
Demombynes, G. et al (2013) Challenges and opportunities of mobile phone-based data collection: Evidence from South Sudan