The challenges of humanitarian information and analysis: evidence from Yemen

By the end of 2018, the UN believed that the situation in Yemen had deteriorated to the point of possible famine. Estimates for December 2018 noted that 15.9 million people (53% of the population) were facing severe acute food security – Integrated Phase Classification (IPC) Phase 3 or higher, despite ongoing humanitarian assistance. Famine analysis in Yemen is conducted nearly exclusively through the IPC, and so maintaining and operating a current-sta­tus needs assessment system through the IPC is a high priority. But the IPC and other humanitarian in­formation and analysis systems in Yemen face significant challenges. This article summarises some of those challenges, based on a recent case study of famine analysis in Yemen. The full study report including methodology can be found at  https://fic.tufts.edu/wp-content/uploads/2019-Evidence-from-Yemen-final.pdf

 Data challenges and constraints

Data transparency, data sharing and independent checks

A major concern regarding the analysis of food secu­rity, malnutrition and famine in Yemen is around data transparency. Most data is collected either by, or in close collaboration with, authorities (either the internationally recognised government in Aden or the de facto Houthi authorities in Sana’a). Data on food security, nutrition and mortality cannot be taken out of the country, and there are ex­treme limits on the extent to which data is shared even within the country. Independent, routine checks are limited or not allowed on certain types of data. In the absence of data sharing, there are questions about its quality and independence. In many cases, data is missing or very limited. SMART surveys are undertaken only very occasionally and are often out of the timeline of IPC analyses. Data is often not available or is as much as a year old by the time analysis is conducted. Data on sectors other than food security and nutrition is limited.

Data quality

Data on nutrition for the November 2018 analysis had been collected as much as a year earlier, with the most recent data collected in March 2018. Data on humanitarian food assistance is con­stantly changing, meaning that results for the impact of food assistance also change. Many analysts believe that data on mortality badly under-estimates actual death rates. Working with two separate authorities complicates data collection.

Early warning and hotspots

Gaps in geographic coverage and different levels of coverage make it difficult to identify ‘hotspots’ or emerging areas of concern due to a rapid deterioration in humanitarian conditions. An initiative from the Nutrition, Food Security and Agriculture, Health, Water, Sanitation and Hygiene and Protection clusters in Yemen identified 107 districts for closer monitoring, but none has been added or re­moved since 2018.

Analysis challenges and constraints

The central conundrum of the analysis in Yemen is that indicators of food insecurity have looked very severe for a long time but malnutrition figures have stayed fairly low, and official mortality figures are very low – even zero in some cases. The main question concerns what could explain nutritional resilience in the face of such a serious, widespread and long-lasting food security crisis. Most of the other causal factors that might be expected to explain the nutrition figures (health, WASH) are also bad. Aside from the fact that the nutrition data was out of date for the 2018 analysis, no comprehensive explanation emerged.

Analytical process

Two different analysis processes – in Aden and Sana’a – are necessary before a national analysis can be completed. The analysis is based mostly on food security data and nutrition teams have their own analytical meetings and are often not involved in the IPC analysis. This compounds the central conundrum noted above.

 Technical capacity and participation

Although the IPC analysis has been conducted in Yemen for the past five years, as is often the case in many fam­ine-risk countries turnover in the personnel involved in the analysis is high. The Yemen analysis in 2018 was the first time that updated guidance IPC Manual Version 3.0. was used, which introduced very different means of doing projections, and different people in the process interpreted the changes differently. Participation in terms of numbers is reported to be good, but local NGOs felt intimidated by the process. It is not entirely clear that the authorities, particularly in Sana’a, trust the process, viewing it as outside their control. At the same time, several respondents noted that there is no verification or voice independent of the author­ities. As a result, judgements about the independence of the analysis depend very much on the perspective of individual stakeholders.

Causal analysis

Finally, there is the question of what is being ana­lysed. Most IPC analysis is concerned with current status outcomes for food security and malnutrition (and, in theory, mortality). In Yemen, however, mor­tality data is often missing, and food security and nutrition data is only about outcomes. There is little specific analysis of causes, in particular conflict, even though conflict is clearly the major driver of the humanitarian crisis. The situation is much the same for other information that is not col­lected at the household level. For example, much was made in the analysis of the strong social links among Yemenis, with the resulting observation that sharing resources – including food – provides a strong if informal safety net that mitigates much of the negative impact of the food crisis. Yet there is little in the way of data to support this claim.

Influences on food security analysis in Yemen

Independence of data collection and analysis

Although many respondents noted that data col­lection processes had improved in recent years, a number of constraints on the independence of the analysis remain. Nutrition data is viewed as very political. Examples were cited where SMART surveys and enumerator training were disrupted by national security officials, making further collection and assessment of information very difficult. Incidents were reported where ‘minders’ accompanied field teams and told people how to answer questions. Combined with concerns about the lack of data transparency and sharing, this has led to a situation where many respondents suspect the independence of the data. While some respondents suggested that pressure from the authorities is at the root of the issues, others blame it on the limited courage on the part of the humanitarian community and in particular, a fear of some of the major donors, including Saudi Arabia and the United Arab Emirates, who are major actors in the conflict.

In terms of the analysis, disagreements have been reported on how final numbers of people in need are determined, but no clear, overall pattern emerges from the interview evidence for this study. Some respon­dents suggested that numbers might be inflated to attract more resources; others suggested that numbers are downplayed to avoid annoying one party or another. Numbers can be downplayed and exaggerated at the same time, as Figure 1 shows. A ‘right-skewed but truncated’ population distribution clas­sifies increasing numbers in IPC Phases 2, 3 and 4, with very high numbers in Phase 4, but no one in Phase 5. There is no a priori expected distribution of population across phases, but this is a highly unlikely distribution.

Figure 1: ‘Right-skewed/truncated’ distributions of populations of selected districts, by IPC Phase Classification, Yemen, 2018

This type of distribution at once highlights – and per­haps over-estimates – the number of people in crisis and emergency, while indicating that no one is in Phase 5, or famine conditions. Nearly half of the districts (158 out of 333) in the most recent Yemen analysis showed this kind of (highly improbable) distribution of population, with increasing proportions of the population respectively in Phases 2, 3 and 4, and no population whatsoever in Phase 5. Furthermore, these figures were for a scenario in which there was no humanitarian food assistance.

Other forms of influence were more subtle. Agencies directly involved in data collection and analysis were extremely careful about what they said in public, effectively amounting to self-censorship. Failure to do so could make access more difficult for future assessments, resulting in difficulties in registra­tion, authorities withholding visas or work permits or perhaps even the expulsion of agencies.

Access constraints

The second major way in which the results of the analysis are potentially distorted concerns popula­tions that are accessible and those that are not. An estimated 1.4 million people are living in inaccessible areas, and the extent to which available data accurately reflects their conditions is not known. Obtaining the necessary permissions to collect data can be very time-consuming due to security concerns or bu­reaucratic constraints, such as lack of coordination between different levels of government in granting permis­sions. Access constraints may be driven by concerns for the physical safety of the enumeration teams, may result from attempts to distort what the data shows or may simply be a result of bureaucratic obstacles.

When this results in missing information, analysts face three choices: extrapolate from out-of-date data (collected when access was possible); use data that is believed to be biased (such as extrapolating from accessible areas); or simply delete inaccessible areas from the analysis (leave them blank both in terms of numbers and mapping classification). All three of these choices have consequences for the independence and quality of the data, and the accuracy and validity of the analysis. For the most part, even inaccessible areas are still classified, but it is not always clear to users on what basis classifications are made.

Influences on the process

Several respondents reported instances where they knew that data had been deliberately manipulated, albeit more likely for the purposes of ensuring resource flows than to influence assessments of the severity of the crisis. Others noted that the issue was not so much about the actual numbers being changed, but that constraints on access, refusal to share the data, the banning of some surveys and the use of others to extrapolate to unreachable areas and difficulties in cross-checking, all meant that the door was open to all kinds of influences on – and varying interpretations of – the evidence.

Finally, there is the issue of how the ‘techni­cal consensus’ is formulated. Several respondents referred to the consensus being driven by the ‘loud­est voice in the room’. A ‘consensus’ outcome is essentially driven by the most powerful individual members of the analysis team.

 Lessons and recommendations

Data concerns. A clear and urgent issue regards data transparency and data sharing. Missing data, data that is extremely out of date or data that is not representative of the specified unit of analysis all constitute significant challenges to rigorous and independent analysis of food security and nutrition in Yemen. Data on mortality in particular is frequently missing. Better early warning information is needed to help identify hotspots where resourc­es (both for assessment and response) can then be concentrated. As a result, the humanitarian community has some major decisions to make related to advocacy for good-quality and optimum coverage of evidence collection and develop strong protocols for data transparency and data sharing.

Analytical concerns. Yemen presents an analytical conundrum that so far has defied full explanation: extremely high levels of reported food insecurity, the collapse of the public health system, a WASH-relat­ed crisis – and yet low levels of reported malnutri­tion and extremely low levels of reported mortality. The fact that this conundrum remains unresolved, and that there are so many counter-narratives, undermines faith in the analysis. Confusion persists between current status (empirical) and early warning (proba­bilistic) or between current status reporting and pro­jections. In addition, the periods between analyses are often very long, and too long for trend extrapolation to provide re­liable results for decision-making.

If the IPC is to be the sole measure of classification for famine, then all agencies involved should ensure that the analyses are more frequent (ideally two or three times per year in a crisis of this severity and magnitude) and more timely in terms of data analysed (no more than two or three months old is the usual standard), the risk of false negatives is significantly reduced and projections are of much greater quality. A more flexible approach to the timing and cover­age of each analysis is also needed. This will require the mobilisation of and support from the highest manage­ment level of agencies and their full support for the IPC process.

Influences. The data collection and analysis process may be influenced in several ways. One of these is access and, when access is blocked, how agency leadership can take up concerns with the author­ities. Some respondents mentioned intimidation as a real deterrent to this kind of support. At the same time, there is persistent pressure, at least at a high level, for positive publicity from donors which are also direct belligerents in the war that is driving the humanitarian crisis. Some agency directors, and at times the Humanitarian Coordinator, engage with the Yemeni author­ities to ensure access. This process needs to be regularised, and pressure needs to be maintained until better access is achieved. This requires strong and sustained advocacy with the authori­ties. Donors can help as well.

Many institutions of government are under the control of de facto au­thorities in Sana’a, and in some cases technical staff have been replaced with political appointees, result­ing in a loss of technical capacity and independence. Continuing to build strong tech­nical capacity is one of the safeguards against undue influence on the process. The politics of information may differ from one period to the next, calling for vigilance and mitigation of the factors that influence the analysis through a sys­tem of governance that is as transparent, participatory and inclusive as possible. The arguably unique political environment in Yemen is testing the limits of the IPC governance system. More attention to governance of the system is required at the most senior levels of the UN and donors in Yemen.

Numerous incidents were reported of ‘the loudest voice in the room’ swaying the analytical consensus. The very high proportion of districts analysed as having a ‘right-skewed but truncated’ distribution of population by IPC phase classification is strongly suggestive of this phenomenon. This results in funding decisions having to be made in the absence of reliable assessment results. The further potential result is that resources are not tar­geted impartially, undermining the very purpose for which these data collection and analysis processes were invented. Building broader partici­pation and an empowered multi-stakeholder analysis without fear or intimidation are probably the best guarantees of independent analysis, particularly if they are strong enough to mitigate potential sources of influence.

Lindsay Spainhour Baker is Research Associate at the Centre for Humanitarian Change, where Peter Hailey is the Founding Director. Jeeyon Kim is Senior Researcher for Resilience at Mercy Corps International. Daniel Maxwell is the Henry J. Leir Professor in Food Security at Friedman School of Nutrition Science and Policy, and Research Director at the Feinstein International Center at Tufts University.