Can joint evaluations promote ongoing collaborative action by NGOs?

In October 2005, something unusual happened in Niger. Staff in the country offices of CARE, Save the Children, Catholic Relief Services (CRS) and World Vision got together to conduct a joint evaluation of their agencies’ responses to the food crisis. None of the country offices had done joint evaluations before, and their decision to undertake one stemmed, not from a request by donors, but because they wanted to learn from each other, and contribute to knowledge in the wider humanitarian sector by making their findings public.

This event might be unremarkable were it not for the typical lack of coordination among NGOs as a whole. Coordination can be difficult for a variety of reasons, including normal competitiveness and the culture of independence so deeply ingrained within the NGO sector. However, it can lead to the uneven distribution of aid, and be confusing for beneficiaries, who receive very similar goods and services from different agencies. Such independent action by NGOs has created a poor public image for the aid enterprise, and led to missed opportunities for advocacy, the sharing of resources and information and more efficient response. As a result, numerous evaluations have stressed that NGOs must get better at coordination.

Collaborative efforts are central to the work of Emergency Capacity Building (ECB) Project, which supported the joint evaluation in Niger. The project is a two-year collaborative venture between World Vision International, Save the Children-US, CARE International, Oxfam-GB, Mercy Corps, the International Rescue Committee and Catholic Relief Services. In addition to the evaluation in Niger, some agencies involved in the ECB have undertaken joint evaluations in tsunami-affected countries, and an evaluation is currently under way in Guatemala. The joint evaluation in Niger offers by far the most detailed information on the value of joint evaluations for ECB agencies. As such, it will provide much of the evidence examined here.

The Niger joint evaluation illustrates the call to collaborate:

The team noted that the [IWG] Partners were reactive in regard to interagency coordination such that the lack of an effective NGO forum in Niger is a serious weakness in effective programming…opportunities were lost for recognition of Partners’ comparative advantages, establishment of joint advocacy positions and for peer training. The latter would have been invaluable prior to the emergency intervention.

Joint evaluations and the coordination process

In Niger, the evaluation helped the NGOs involved to:

• Hold regular coordination meetings:CARE and Save organised bi-weekly meetings of NGOs in October to share quantitative and qualitative reports, and to discuss the scaling-up and targeting of nutrition activities. These meetings are now periodic, and are led by different NGOs on a rotating basis. They have helped NGOs lobby the government of Niger and the World Food Programme (WFP) for support in specific areas, especially regarding nutrition. In addition, the agencies credit the joint exercise for prompting them to set up an NGO coordination forum in Niamey, to which more NGOs have been invited.

• Expand their partnerships with UN agencies:Following recommendations from the evaluation, CARE and CRS are working with WFP to align their emergency rehabilitation goals and activities, especially through implementing Food-For-Work activities and stocking cereal banks with WFP resources. Several NGOs, including Save and World Vision, are meeting WFP and UNICEF to coordinate a new round of food distribution. Discussions include how to work together and support other partners in areas where they are less able to distribute food, and how to come up with a set of common indicators to identify zones for targeting.

• Increase their voice at national coordination meetings:By working together to develop a unified national action plan for nutritional recuperation, NGOs are able to present a more unified message at national health and nutrition coordination meetings, originally sponsored by UNICEF and the World Health Organisation, but now managed by the Ministry of Health.

A more global perspective

In Niger, the agencies were able to collate their different perspectives, and this gave them a deeper understanding of the crisis and its causes. The joint evaluation was about ‘understanding the crisis, its long-term chronic nature, as well as agency responses, our impact and where to go with recovery efforts’, notes one colleague. The evaluation also found that each agency had its own strengths; by working cooperatively, the four agencies are likely to have a greater impact in their emergency and development programmes. Such a finding would have been difficult to discern had each agency done an independent evaluation.

Shared technical expertise and local knowledge

Increased communication with partners has allowed for greater sharing of technical expertise and local knowledge, in addition to helping agencies learn about one another’s programmes. The Ministry of Health and Médecins Sans Frontières (MSF), for example, helped train CARE agents to identify various types of malnutrition and to treat moderate malnutrition. In the Zinder region, CRS trained staff of the NGO GOAL in the implementation of voucher fairs. The government, Save, CARE, CRS, World Vision and several local NGOs worked together on food distributions, which were coordinated at weekly WFP and government meetings from August until October.

Setting realistic and shared goals

One of the objectives of the evaluation was to assess the short-term impact (outcomes) of the four agencies’ emergency response work. The agencies defined different goals for their emergency programmes, including to save lives, to improve food security and to strengthen the economy. The evaluators noted that most goals were not met, or were only met temporarily. What the evaluators did not explore is how realistic and measurable any of these goals are for an emergency food distribution programme.

By sharing the goals they set for their programmes, the agencies can begin to assess whether they are realistic, and joint processes can help them to agree on a common set of more achievable and measurable goals, and develop methods to evaluate them. A review of the multi-agency evaluation report indicates that the most likely goals they could achieve in chronic food emergency situations are:

1. To reduce the food insecurity of households affected by drought and locust infestation.
2. To protect the nutritional status of specific vulnerable groups (i.e. pregnant and lactating women, children under five years of age and the elderly).
3. To enable affected households to preserve their asset base.

Helping agencies increase transparency and accountability

Individual agency evaluations are often internal documents that never make it into the public domain. However, both the tsunami evaluation and the Niger evaluation have been posted publicly on the ALNAP website and on the ECB Project’s website. This move is significant: by opening up both the favourable and less flattering appraisals of agencies’ performance to public scrutiny, agencies are increasing their transparency and accountability to their stakeholders. The evaluation process itself also allows the agencies to hold one another to account through peer review. In Niger, the four agencies involved in the joint evaluation have called for a follow-on joint event to review progress against the recommendations of the evaluations and to strengthen inter-agency collaboration.

Conclusion

The preliminary evidence suggests that joint evaluations set a precedent for greater collaboration, sharing of resources and information and learning among agencies, and provide them with the opportunity to develop trust, and to regard one another not as competitors but as partners. It is also clear that success is highly dependent on the process itself, and the level of on-the-ground engagement in the evaluation exercise. For example, whether agencies meet and do some analysis together and form relationships as they identify commonalities goes a long way to determining the benefits of the evaluation. A highly skilled team leader who knows how to set up short analytic interagency discussion meetings to agree the findings, conclusions and recommendations of the process with agencies on the ground will ensure that the evaluation report is owned by the teams in-country, and that the recommendations will be taken forward.

Individual evaluations are useful for examining the complexity of logistics, monitoring, human resources and financial management, and the extent to which regular programme activities mitigate crises. These aspects are important, but they often distract agencies from focusing on the larger question of how collaboration, coordination and advocacy can improve NGO responses before, during and after a crisis. Joint evaluations enable agencies to understand the collective contribution they make to mitigating the effects of the crisis. This is important because so many agencies often respond together, and trying to attribute impact to any one intervention is difficult. Joint evaluations also help agencies to keep their focus on these larger goals, and enable them to form collaborative relationships so that NGOs can collectively improve the quality of their response.

Malaika Wright is Knowledge Management/Research Officer with the Emergency Capacity Building Project. Her email address: is mwright@care.org. Pauline Wilson is Accountability and Impact Measurement Initiative Manager, Emergency Capacity Building Project. For more on the Emergency Capacity Building Project, visit www.ecbproject.org.