Does accountability deliver results?

Accountability to affected populations. The ubiquity of the phrase in the humanitarian sector masks a crucial fact: while formulating a policy statement detailing accountability aspirations is relatively easy, actually being accountable to recipients of aid is often, surprisingly, difficult – and demonstrating that you have been accountable is more challenging still. Part of the problem is the sheer diversity of accountability delivery methods available. But more problematic is the fact that, despite the decade-long focus on accountability, little research has been conducted on the link between accountability mechanisms and programme effectiveness. As Paul Knox-Clarke of ALNAP has suggested, arguments for accountability mechanisms ‘sound as if they ought to be true. Perhaps this is why, over the years, we have done so little to investigate whether they are true on the ground’. Andy Featherstone, Improving Impact: Do Accountability Mechanisms Deliver Results?, Christian Aid, Save the Children and the Humanitarian Accountability Partnership, June 2013, http://www.hapinternational.org/pool/files/improving-impact-do-accountability-mechanisms-deliver-results.pdf

HAP Standard as the benchmark

The Humanitarian Accountability Partnership (HAP) joined Christian Aid and Save the Children UK, with methodological support from ALNAP, in an attempt to address this gap and generate evidence of the causal link between accountability mechanisms and programme quality. Acknowledging that no study of this size could provide a dataset solid enough to demonstrate a global trend, the methodology was designed to be open-source and replicable in anticipation that other organisations would grow the body of evidence across a broad spectrum of programme sites and contexts. Indeed, since the initial study of projects in Kenya and Myanmar, Save the Children UK funded an additional study of its Disaster Risk Reduction programme in Sindh province in Pakistan, with a particular focus on children, the results of which are included in this article.

The accountability mechanism tested was the 2010 HAP Standard in Accountability and Quality Management, in particular the three benchmarks (reference points against which performance can be assessed) that most closely relate to community engagement in project planning and implementation: sharing information, participation and handling complaints.

The approach

The test sites were located at partner projects of Christian Aid (in Kenya) and Save the Children UK (in Myanmar). The Christian Aid project worked with community-based organisations to help identify the main issues preventing them from establishing and maintaining a good standard of living. The Save the Children UK project focused on child protection and non-formal education.

The first step in the research was to assess whether the three targeted accountability mechanisms were rolled out effectively in the country programmes under study. This was done using an adapted version of the Listen First Framework, rating performance on four levels: from basic to intermediate, to mature, to HAP compliant. In both Kenya and Myanmar, the three benchmarks in question were all rated as either intermediate or mature. The second step involved assessing the contribution of the three accountability mechanisms to programme quality. For the purposes of the research, ‘quality’ was operationalised using the Organisation for Economic Cooperation and Development (OECD) Development Assistance Committee (DAC) criteria for evaluating aid projects: relevance, effectiveness, efficiency and sustainability. All assumptions around the impact of accountability mechanisms were linked to the four DAC criteria and assessed from the perspective of the communities targeted by the projects by an independent consultant not associated with the aid agency.

Accountability is worth the effort

Andy Featherstone, the author of the resulting study, Improving Impact: Do Accountability Mechanisms Deliver Results?, summarised the findings as follows: ‘A modest investment in information sharing (in terms of financial resources, staff time and agency commitment), involvement by project participants in the design and delivery of programmes, and ensuring there is a means of listening to and acting on feedback, brings a significant return – not only in participant satisfaction and engagement in projects, but also in the tangible success of projects’. Improving Impact: Do Accountability Mechanisms Deliver Results?

One of the most valuable findings from the research was that accountability is not just a Western concept, foreign to communities where many aid agencies operate. The discussions with local communities highlighted the fact that the link between accountability and programme quality is in fact very clear to the people targeted by aid organisations. In the words of a community in Kenya, ‘Before [the partner organisation] came, other programmes have failed because they lacked accountability and there was corruption. Accountability is a key part of the success of the programme’.

When it came to the contribution of accountability mechanisms to quality as measured by the four DAC criteria, some links were more obvious than others, though clearly they all, to some extent, depended on contextual factors.

With regards to relevance, participation was considered to have positively contributed to the successful targeting of beneficiaries, and was seen as an important component in ensuring that the project focused on the needs and priorities of communities. In Pakistan, participation ensured that the priorities of the whole community had been met, including more marginalised members. The Pakistan study found that the greatest impact of participatory mechanisms was on the relevance and appropriateness of assistance, which contributed to a greater sense of ownership and more sustainable programmes. Less frequently, complaints handling mechanisms were mentioned as catalysts to adapt projects to better meet local needs and contexts. In Pakistan, participants said that they had contacted Save the Children directly over issues of quality of materials and timeliness of work, which led to an improvement in programme quality.

In terms of effectiveness, the clearest link to the accountability mechanisms was the contribution of information sharing in generating participant understanding and uptake of the project. The study also demonstrated that information sharing built greater trust between participants and the organisation. Some specific examples were also provided to show how accountability mechanisms respect the dignity of participants and empower communities, or help identify and address problems swiftly (including fraud and mismanagement). However, no evidence was found of the accountability mechanisms in question strengthening the security of aid providers; neither in Kenya and Myanmar, where security was good, nor in Pakistan, where security was much more of a concern. This indicates that, though there were examples of transparency and good relationships improving security, this is not a given in every context.

As far as efficiency is concerned, the research had less to offer, though there were several examples of how better project ownership by communities led them to actively check and challenge the quality of work conducted by contractors, provide more cost-effective options for procurement or raise instances of fraud, thereby improving value for money. In Pakistan there was evidence of an improvement in efficiency as a result of the complaints and response mechanism. In one example a delivery of poor-quality bricks was reported by mobile phone and arrangements were made for their replacement. This led to an immediate improvement in quality and hence a more efficient use of resources.

The sustainability of programmes was most commonly linked to the use of participation in project design and implementation, which resulted in stronger ownership. Less frequently, accountability mechanisms as a way to improve relevance were also cited as contributing to sustainability.

Finally, and perhaps most interestingly, both the Myanmar and Kenya case studies offered a number of examples of how exposure to, and use of, accountability mechanisms had influenced the actions of communities in a variety of ways that went beyond the expected outcomes of the project. These ranged from communities adopting decisionmaking processes experienced in NGO-led projects in their own social organisation, through to feeling empowered to raise complaints with other actors, for example banks. While it is difficult to quantify this influence, in each circumstance the community made explicit reference to the accountability mechanism as a primary sway.

These results offer a credible snapshot of the impact of accountability mechanisms in three distinct contexts, and give the clearest indication yet that accountability rhetoric is paying dividends in work on the ground.

What we learned on the research side

The methodology was designed to apply to a variety of situations and, so far, it has proved suitable for all three programmes assessed. However, it is likely that some of the DAC criteria would not be as relevant to rapid-onset emergencies, and the methodology would need to be contextualised in such cases. Additionally, striking the right balance between a solid methodology ensuring appropriate representation and a study that was cost-effective and could be conducted rapidly was a challenge. To make the process quicker, the methodology was revised for the pilot in Myanmar, allowing different groups to be involved, and this made for a quicker process while ensuring representativeness. It is important to acknowledge that, because projects were selected where participation was good, there was already a culture of discussion about accountability mechanisms. Similar discussions would be more difficult in instances where such dialogue had not already taken place.

One issue that did arise was the interplay between formal and informal accountability mechanisms. In Myanmar, the community was more comfortable using an informal mechanism for feedback. In Kenya, by contrast, a very formal mechanism was considered more robust and more highly valued. This demonstrates that accountability mechanisms need to be closely tailored to the community that they serve – one size does not necessarily fit all. It also reinforces the HAP guidance that communities should be consulted when accountability mechanisms are designed.

Perhaps the biggest drawback of the research was the absence of any counterfactual – a project with no accountability mechanisms in place. Primarily this was because those projects studied were undertaken by the same partner organisation, and the main difference with supposedly weaker mechanisms was that projects had been running for shorter periods of time and mechanisms had not yet been fully rolled out. This makes the prospect of trialling the mechanism in a humanitarian setting attractive since there is a greater likelihood that a counterfactual will be found.

Currently there is significant momentum behind the accountability movement; this is visible in the response to Typhoon Haiyan in the Philippines, where the profile and on the ground capacity of communication with communities, complaints handling and participation is much higher than in previous large-scale disasters. To support this and ensure that affected populations remain at the heart of what the sector does, research needs to provide more detailed evidence and understanding of the value of accountability, and encourage this type of assessment to be part of the range of external evaluations that routinely look into the quality of aid programmes. Only when those who are meant to benefit from aid programmes are systematically involved in their monitoring and evaluation can the sector truly claim to be accountable.

Murray Garrard is Communications Officer at the Humanitarian Accountability Partnership. Improving Impact: Do Accountability Mechanisms Deliver Results? is free to download from the HAP website at http://www.hapinternational.org/what-we-do/research-and-learning/current-research-projects.aspx, together with the detailed methodology. The Save the Children UK study in Pakistan will soon be available from the Save the Children UK website (http://www.savethechildren.org.uk/) and will also be available in HAP’s Quality and Accountability Resource Library (http://www.hapinternational.org/resources/resource-library.aspx). Those wishing to use this methodology to conduct further research may contact HAP’s Head of Policy and External Relations, David Loquercio, for additional advice, on dloquercio@hapinternational.org.