Real-time evaluations in Darfur: some suggestions for learning

Several real-time evaluations (RTEs) have been carried out in Darfur and Chad since the start of the crisis there. Three were presented at the biennial meeting of the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) in December 2004, and others have also been conducted over the past months. This article looks at some of the issues arising from these evaluations, and suggests areas where some valuable lessons might be sought.

RTEs first appeared under that name at the end of the 1990s. There is no generally accepted methodology for RTEs, but in practice they are carried out rather like standard evaluations. Although they are in some ways dissimilar to ex-post evaluations, they share with them an emphasis on accountability and learning. However, they tend to focus less on impact and more on performance, with a view to seeing how this can be improved through immediate feedback to programme managers. RTEs also provide an opportunity to identify not only current (i.e. real- time) issues, but also issues that will possibly be of interest in any later evaluation.

Although RTEs have become fashionable among several agencies, in Darfur it is likely that agencies turned to them to help them understand what they felt was not a very satisfactory response, and to help them improve it. It is clear that, in coming years, the Darfur experience is likely to be referred to as seminal or paradigmatic. This article therefore draws on the author’s own experience of doing RTEs in Darfur and Chad to explore the issues that seem to have emerged, mainly with a view to suggesting areas that might be of particular interest for later reflection or evaluation within the humanitarian community at large.

Timing

There is a widespread belief that the individual and collective response to the humanitarian situation in Darfur was late and/or slow. This is a particularly acute issue for those agencies that had a presence, in some cases for many years, not only in northern Sudan but in Darfur itself.

Several factors explain the relatively slow response. First, the crisis evolved relatively slowly; it did not explode all at once. Second, it was overshadowed by other crises elsewhere, crises that were of equal or greater scale. Iraq dominated international attention in 2003–2004. The international community was arguably also inured to crisis in Sudan. There was – and is – a hope that the North–South peace process will succeed, and it was felt, particularly by the UN, that to distract from that, or to change the dynamics of the pressures on the Sudanese government, would jeopardise that process. In fact, of course, the rebellion in Darfur coincided with progress in the North–South peace process. The failure by international agencies, particularly non-governmental ones, despite the warnings of their local staff in some cases, to recognise the significance of the escalation of the conflict in Darfur, inevitably raises questions as to why they were not more alert.

Once the scale and potential importance of the Darfur crisis started to become apparent, the Sudanese government slowed the response by imposing a range of restrictions on the movement of goods and people. Meanwhile, the perception of insecurity and the lack of clear channels of communication with the rebels limited agency access to territory they held. The impending rains threatened to slow the response still further, though this does not seem to have heightened the urgency to respond. For those agencies that were already in Darfur, the transition from ongoing development programmes to a humanitarian operation seems to have caused problems. The usual tensions, of course, combined with logistical and official impediments, made this a slow process.

Even without bureaucratic blockages and security concerns, Darfur poses significant logistical challenges: it is far from Khartoum, and Chad’s border with Sudan is remote from the rest of Chad. It took a long time to assemble sufficient vehicles and communications equipment to enable agencies to feel safe enough to venture into these undeveloped areas. While some vehicles could be hired locally, these were too few and usually not considered reliable enough for safety and security reasons. Finally, it took a very long time to hire local as well as international staff.

It is likely that we will need, not only to explain why we were slow or late, but also to seek ways around such obstacles in the future, as well as institutional methodologies to ensure that slow, obscure or endemic crises do not get overlooked. Agencies may have looked at the blockages internal to themselves, rather than at how they might have navigated external constraints. Agencies with longstanding programmes that did not adapt rapidly enough to a humanitarian mode will benefit from understanding why. It will certainly not be good enough to put the prolonged failure to protect the people of Darfur down to unavoidable circumstances, or factors that could not be circumvented.

Accountability to affected people

We all know that accountability to affected people is difficult, particularly where there is violent conflict. While there have been some excellent interventions in support of the public health and nutritional needs of some people in Darfur, and there is a widespread culture of consultation among at least some agencies implementing them, this tends to be instrumental, rather than collaborative or supportive. It also relates only to the field programme; advocacy elements do not engage with people’s aspirations beyond their humanitarian needs, and the immediate causes of those needs. External agencies often rely on local organisations as ‘partners’ to be an access channel to those they seek to assist and protect, or even as a proxy for them. In this case, there are very few local organisations. If agencies have had other recourses, these should be recorded and their efficacy should be looked at. If there is no valuable material to learn from here, then that itself is a challenge that must be explored.

Depth of staff experience

Many of today’s most experienced humanitarian workers have done stints in Sudan, including in Darfur, before. This depth of experience was needed in situ in Darfur, but few of those people are still front-line programme managers, or have lives that would allow them to spend months so far away from their homes. There was also a general dearth of managers able to deal with the complexity of the operations, organisation and politics of Darfur. What then are the lessons for human resource mobilisation? Or for institutional learning or training? Or for management support?

Local and international

In October 2004, there were, according to OCHA, a reported 800 expatriate staff and 5,000 local staff in Darfur. These numbers are enormous, and have no doubt risen since then, though the relative proportion of international to national is not exceptional. My impression, albeit with a fair degree of confidence, was that very few senior agency staff were Sudanese.

It is not clear what ‘local’ means in this context: staff from other parts of Sudan, including the South, are only to a limited degree local, and staff from other parts of the North are unlikely to speak the local languages and may be tainted with the suspicion that they are politically aligned. Encouragingly, many staff from many different parts of the world were present, including a large number from neighbouring countries.

Arguably, the staffing situation in Darfur is unlikely to be replicated elsewhere, but it is nonetheless fascinating to see the traditional expatriate–local dichotomy so comprehensively compromised and yet still so strong. This is worth looking at to see what factors can reinforce or break down this bane of emergency response.

Scaling up

The reach for scale among agencies was enormous and determined. However, Darfur’s marginalisation – one of the causes of the rebellion – meant that, in practice, there were no large numbers of educated people to recruit, and government agencies had very limited resources. The government blocked recruitment from Khartoum (though only a limited number of people would want to move to the far reaches of the west of Sudan anyway), and administrative obstacles added to Sudan’s already complex and process-heavy labour laws. Once recruited, new local staff need basic training before they can become effectively operational.

The aspect of this problem that will bear future reflection is how to expand a programme and an organisation significantly and at speed, while maintaining the identity of the humanitarian organisations, and of the humanitarian enterprise and its standards, quality and principles. Many agencies tried very hard to bring in large numbers of local staff, but these individuals had little or no experience or knowledge of humanitarian practice. It would be helpful to know what methods for doing this appeared most effective, what the key stumbling-blocks were and whether any tools developed in Darfur could be more widely used.

Coherence, advocacy, protection and field programming

The two dimensions of this ‘crisis of protection’ are that violence and coercion have caused the humanitarian need, and that people continue to suffer from violence even after they have taken refuge, and are to some extent under humanitarian protection. Agencies’ advocacy work has encompassed both of these elements. The field paradigm for the international response had been ‘protection by presence’, a form of substitution for a government that is failing to protect its people.

As is often the case, some field staff were concerned about whether advocacy messages were sufficiently connected to field programmes. They were simultaneously struggling to turn the common protection analysis into programmes that would serve to protect. This was a problem for a large part of 2004, and perhaps beyond. It would be helpful to ensure that lessons about how to fashion programmes that both include relevant advocacy and provide protective assistance do not remain within the ‘protection community’, but are included in the evaluation of assistance programmes.

Coverage, impartiality and the numbers game

Two years on from what could be conceived of as the beginning of this crisis, access to all those affected is still, I believe, far from complete. As recently as April 2005, a British parliamentary report suggested that twice as many people had died as had been previously thought. This has been challenged, but there are no authoritative figures. The humanitarian community claims to be serving large numbers, yet coverage is patchy and the impartiality of the overall effort must therefore be called into question.

As we mark the tenth anniversary of the Red Cross/NGO Code of Conduct, we should perhaps reflect on our ability to live up to the articles within it, specifically in this context Articles 1, 2 and 3 that enjoin upon us impartiality and neutrality.

Anything you do can be seen as political

Sudanese governmental support for the wider humanitarian effort is variable, and this variability often seems hard to understand or explain. Agencies commonly suspect or distrust the government, either in its motives or in its intentions. Equally, the humanitarian community in Darfur is suspect in the eyes of the government and many Sudanese people – of acting as the vanguard of a US invasion, as religious proselytisers or intent on arranging the overthrow of the current government. Loud external criticism does nothing to inspire trust in the purity of our motives, particularly in the context of a political society that does not have a great deal of experience of democratic openness and is not likely to find it easy to shake off its semi-pariah status internationally. Nor does the arrival of large numbers of expatriate staff, some of whom lack personal qualities or professional qualifications to justify their presence. In the uneasy balance between humanitarian and political attitudes to the crisis, the UN is compromised by the resolutions it passes condemning and putting pressure on the Sudanese government.

We all have views about the compatibility or otherwise of political and humanitarian motives and actions. This situation seems to me an important subject for discussion and deliberation. The argument that humanitarian action should not be political does not stand up in Darfur.

Chad: contextual analysis and working with local authorities

Most of the above is about Darfur and Sudan, rather than Chad or Darfurians in Chad. The Chad programmes have always been the poor cousins, and rightly so, but they also have their own particular lessons. Among these are why the contextual analysis from Darfur seemed to be so little known in Chad, at least in the middle of 2004. A first guess is that this was for institutional reasons, but there are certainly lessons for agencies about how we collectively frame and share contextual analysis.

The other observation is that there was little contact between those programming for the refugees in Chad. Local authorities’ compliance was generally being assumed, local leaders’ role was all but invisible in practice, although often given lip-service, and local communities’ acceptance of the presence of the refugees was acknowledged while the dire consequences of it being eroded were noted.

If these observations are even partially correct, then there is much to be thought about in relation to the humanitarian community’s respect for the country in which it is operating; in relation to its self-centredness in seeing itself as overly important; and in its short-sightedness regarding the best ways of ensuring the best outcomes for the refugees, rather than for its own activities.

Evaluation

The Darfur crisis has been extensive geographically, in scope and in time. The response to it has taken a great deal of effort and money, as well as attracting justifiable interest within the humanitarian sector. It will undoubtedly attract a great deal of further evaluation. How this can be coordinated, and energies not duplicated, is something that should be addressed sooner rather than later. The issues that have been mentioned above are certainly not comprehensive, but they might form the basis of a shared set of issues of concern that agencies can address together. This approach, rather than an individual agency focus on its own achievements and failings, would indicate more principled, accountable practice in relation to those affected by this crisis. It might incidentally give added value from real-time evaluations, and therefore constitute an addition to the discussions about whether RTEs can be of any real value to programming agencies.

Maurice Herson is an independent humanitarian consultant.