Not a Rolls-Royce but it gets you there: remote mobile food security monitoring during the Ebola crisis

The Ebola crisis marked a coming of age for the use of mobile technologies in the humanitarian sector, with food security assessments leading the way. Movement restrictions and quarantines, in addition to fear of contracting the disease, made implementation of traditional face-to-face food security assessments in Ebola-affected communities extremely difficult. The rapid spread of Ebola and concerns as to how the outbreak could negatively influence market access and food availability also created a need for regular updates on food security.

To overcome these challenges, the World Food Programme (WFP) deployed a fully automated, mobile-phone based remote food security monitoring system in Liberia, Sierra Leone and Guinea. The design of the system incorporated lessons from a pilot in the Democratic Republic of Congo (DRC). Jean-Martin Bauer et al., ‘A New Tool in the Toolbox: Using Mobile Text for Food Security Surveys in a Conflict Setting’, HPN blogpost.  Since September 2014, WFP has been implementing monthly rounds of remote data collection through text messages (SMS) and interactive voice response (IVR – pre-recorded audio messages) through GeoPoll. See http://research.geopoll.com.  Phone surveys are sent to between 500 and 1,100 randomly selected respondents in each country. In line with best practice, these surveys comprise short, simple questions that require straightforward responses, namely food prices and information on how households are coping with food shortages. In December, an open-ended question was added to the SMS surveys to allow respondents to share their perceptions of food security in their communities.

Mobile technology allowed WFP to set up a basic food security monitoring system in a very short time, less than one month after the declaration of Ebola as a public health emergency. The system delivered information quickly, providing regular updates as the epidemic spread. Beginning in September 2014, WFP began publishing monthly food security reports and datasets detailing changes in households’ coping strategies (the coping strategies index (CSI)) and food prices. See http://vam.wfp.org/sites/mvam_monitoring  Given low cell phone coverage and ownership in the three countries, survey results have some urban, male and wealth biases. However, considering the urgent nature of the evolving epidemic and the lack of alternative sources of information, it was felt that data collection should proceed, and that biases would be accounted for during analysis and interpretation.

SMS and IVR as survey modes: strengths and limitations

Our experience with two data collection modes – IVR and SMS – allowed us to assess their relative performance. People in the Ebola-affected countries either received a series of questions via SMS, to which they would respond by text message, or via IVR, to which they would reply by pressing keys on their phones. The surveys were free to reply to, and respondents received a small airtime credit as an incentive after completing the surveys.

SMS and IVR performed differently in terms of cost and data quality. It was much cheaper to collect data by SMS than by IVR. For the same number of questions, an IVR questionnaire cost $35 to complete, compared to $6 by SMS. The quality of data collected by SMS was also better than for IVR. Figure 1 compares the distribution of the CSI data for SMS and IVR. The profile of the CSI data collected by SMS is close to what face-to-face surveys produce: many responses at zero and a progressively diminishing number of responses for higher CSI values. By contrast, IVR surveys tended to produce a bell-curve distribution, indicating that IVR was producing data that differed from face-to-face surveys.

There was also some indication that IVR surveys were producing higher CSI estimates than SMS. For instance, when we switched from IVR to SMS data collection in November we observed an average 8.1 point drop in the indicator across Liberia. However, in Lofa County, where we had used SMS in October and November, the drop was a much smaller 0.8 points. We were therefore cautious in interpreting IVR results, and moved to SMS whenever possible. A more structured study to evaluate how different survey mode affects results (e.g. SMS versus IVR) is being planned in Guinea. This will help us quantify the extent to which the survey modality affects the responses.

For food prices SMS surveys produced data with fewer outliers than IVR, presumably because the respondent could read and edit responses prior to sending them. Between 60% and 80% of responses collected by SMS required no cleaning, compared to much lower percentages of clean responses for IVR. Tweaks to the SMS questionnaire led to over 90% of responses being clean for Sierra Leone and Liberia in data collection rounds four and five.

Why did SMS achieve better results than IVR? On average, respondents took 18–19 minutes to complete a questionnaire by SMS, compared to six minutes for IVR. This might suggest that the ‘pace’ of an IVR questionnaire leads to greater data quality issues. This supports our theory that SMS is a user-friendly medium of exchange for collecting data remotely, as it allows people to reply at a time of their choosing, read questions at their own pace and review their reply before submitting their answer.

While SMS was cheaper and more reliable, there is scope to use IVR for data collection. In contexts similar to that of the Ebola-affected countries where SMS may not be possible due to technical reasons, IVR could be used as a last resort, or for simpler questions. It is also thought that IVR could have an important role to play in remote surveys in communities with very low literacy levels.

Due to the use of automated data collection modes, thus far we have not successfully administered more complex survey modules like the food consumption score (FCS) as the indicator has proven too cumbersome to be reliably collected through SMS. This meant that we were only able to track how households’ experiences of dealing with food insecurity changed, not changes in their actual food consumption.

Did the data describe reality?

The findings suggested that affected communities were facing a ‘slump’ food access crisis, characterised by low household incomes and reduced demand, rather than scarcity and spiralling food prices. Overall, low purchasing power, rather than food price hikes, constituted the main barrier to household food access: ‘It’s not the price of commodities that is high, but rather the wages are low’, read a text message received in December from a respondent in Sierra Leone. These findings suggest that the consequences of Ebola outbreaks had immediate, indirect and substantial impacts on wages and labour markets. These effects were also reported, in more detail, by other sources. See for example World Bank, The Socio-Economic Impacts of Ebola in Sierra Leone, http://www.worldbank.org.  The data generally showed that food security indicators were poorer in rural locations compared to the capital cities, which were all experiencing Ebola outbreaks. The hypothesis was that greater market access in the capitals allowed city dwellers to cope better with livelihood change than their counterparts in other areas. Our data also suggested that households led by women were generally more food insecure than households led by men. We also noted that more deprived households used negative coping strategies much more often than their better-off counterparts.

Our data also suggested that the areas initially exposed to the Ebola outbreak (Forest Guinea, Lofa County in Liberia and Kailahun District in Sierra Leone) had the highest levels of negative coping, indicating a relationship between zones that experienced high levels of Ebola cases and food insecurity. We further observed that, as the epidemic spread to northern Sierra Leone and western Liberia in November, food-related distress in those newly affected areas also increased. Findings in December suggested that, in places where the epidemic had subsided, Ebola-induced food insecurity remained. It is possible that the Ebola outbreak may have prompted longer-term effects on household incomes and assets, a hypothesis that in-depth needs assessments must consider.

The system was also able to capture seasonal changes in indicators (with declines in coping and food prices observed during and immediately following harvest), matching the International Growth Centre’s assessment of food price trends in Sierra Leone. International Growth Centre, The Economic Implication of Ebola, 2015, http://www.theigc.org.  However, we were unable to pick up the more granular, localised price anomalies reported by the IGC in Sierra Leone, or by Premise in Liberia. See https://data.premise.com/indicators/liberia.

The system was able to tell a story and support discussions on operational response based on changing data trends. However, it was unable to zoom in on specific zones, making it difficult to observe nuances between areas. As such, collected data was of limited use when determining how to target assistance other than geographically.

Discussion: field-ready for other emergencies?

There was no alternative to remote mobile data collection in the Ebola-affected countries due to restrictions on staff movement that limited routine assessment activities. The crisis provided an opportunity to set up a remote data collection system, which was put in place quickly and delivered data cost-efficiently. This experience shows that the tool could provide some added value in other settings where physical access to survey respondents is irregular or otherwise restricted – for instance in conflict. WFP’s work with call centre-based phone surveys in central Somalia and eastern DRC points to the potential of such approaches. See http://mvamblog.wordpress.com.

While the Ebola crisis shows the promise of automated food security monitoring systems, a word of caution is necessary. Remote mobile surveys are technically tricky and labour-intensive. Automating data collection through SMS or IVR is no shortcut: in order to achieve the desired outcome of quick, accurate and inexpensive food security monitoring and reporting, agencies must continue to invest in improving remote data collection techniques, data management and analysis.

Due to its streamlined nature, remote mobile data collection, on its own, is unlikely to satisfy the multiple (and growing) information needs of humanitarian managers. Mixed-mode systems, that exploit the strengths of both face-to-face and mobile data collection and allow for the triangulation of information, would be ideal. However, food security information systems tend to be weakest in the resource-poor environments where they are most needed.

Experience with the Ebola response suggests that humanitarian agencies will have even more access to high-frequency information on developing food crises, perhaps for a broader range of indicators. The challenge will be managing this large amount of information responsibly. Humanitarians will have to develop agile ways to access and exchange information on household food insecurity, ideally coupled with relevant market and health facilities data.

Jean-Martin Bauer is an Analyst at the WFP Analysis and Trends Service. Anne-Claire Mouillez is an Advisor at the Service, and Arif Husain is Chief Economist and Head of the Service. The authors would like to thank Mireille van Dongen, who provided research assistance, and Marie Enlund, Maribeth Black, Silvia Passeri, Tobias Flaemig and Susanna Sandstrom for their comments on this article. WFP’s work with remote mobile data collection is supported by the Humanitarian Innovation Fund and USAID.