Developing systematic feedback mechanisms: the Listen Learn Act project

Before the ink was dry on the December 2014 Copenhagen agreement on the Core Humanitarian Standard (CHS) to improve the quality and effectiveness of aid, one of the key partners in the two-year process that led up to this newly created policy framework proposed creating a pilot project that would put it into practice in the field. DanChurchAid, a Danish humanitarian non-governmental organisation, enlisted Ground Truth Solutions (GTS) and Save the Children Denmark to design a test project with multiple partners in Africa and Asia.

The idea was to use these newly minted humanitarian guidelines and work with international and local humanitarian agencies to measure and improve compliance with the first five of the CHS’s nine objectives – those lending themselves to validation by affected people and trackable using GTS’s Constituent Voice (CV) feedback methodology. The aim was to develop systematic feedback mechanisms to integrate the views of affected people into the design and implementation of humanitarian programmes based on their views of the humanitarian response as appropriate and relevant; effective and timely; that it strengthened local capacities and avoided negative effects; that the response was based on communication, participation and feedback; and that complaints were welcomed and addressed.

Over the next nine months, DanChurchAid, GTS and Save the Children Denmark worked on planning and securing funding for what became the Listen Learn Act (LLA) global quality and accountability project. The project fitted well with the special requirements of ECHO’s Enhanced Response Capacity for projects that increase the resilience of people and communities hit by humanitarian crises, while promoting personal dignity and empowerment. The LLA consortium’s proposal secured funding and LLA was formally launched in September 2015 in four countries: Nepal (earthquake); Lebanon (Syrian refugees); Ethiopia (South Sudanese refugees in Gambella); and Mali (food insecurity and violence in the north of the country).

The consortium worked with four different humanitarian organisations in each of the four countries – a total of 16 groups and programmes – holding workshops to help train staff in the methodology, design and field-test the surveys and regularly collect and make sense of the responses from affected people in the four countries. Three rounds of surveys of 300–400 affected people were conducted in each country. Participating organisations received real-time feedback from beneficiaries and guidance on how to analyse the data and use it as the basis for course corrections. The consortium partners helped guide each organisation engaged in the project on how to use a variety of communication vehicles to disseminate survey results back to field staff and to affected communities. A major component of the LLA project was broader dissemination and education about quality and accountability through a scenario-based LLA Training Course for humanitarian staff, produced in English, French and Arabic.

By the time the pilot project ended in April 2017, some 18 months later, a number of useful lessons had emerged from this novel, systematic approach to tracking and monitoring the perceptions of affected people. Regardless of the type of humanitarian assistance – from psychosocial support to food and shelter assistance – the LLA project showed that continuous, proactively sought feedback can be a useful performance management tool, especially when agencies initiate and sustain regular dialogue with communities throughout the project cycle and use the feedback to make programmatic course corrections. The more conventional process offers affected people complaints mechanisms to raise their concerns, to which agencies must then respond case by case.

Another important takeaway from the LLA project’s fieldwork is that the CV methodology is relevant across a broad range of contexts. According to Andy Featherstone, a humanitarian action and social development consultant and author of the comprehensive learning report on the LLA project, it also ‘encourages organizations to work across silos, bridging different parts of humanitarian agencies – monitoring, evaluation, accountability and learning, operations and management – and in so doing ensures that accountability is a discussion across the whole organization rather than being siloed in a team or departments’. In addition, Featherstone suggests, ‘the CV methodology was considered extremely valuable in promoting the CHS internally within participating organizations and more broadly with peer agencies’. At the end of the project, the local NGOs involved were particularly enthusiastic about incorporating the methodology – or specific aspects of it – into their humanitarian programming work.

One of the more interesting results from the LLA project, according to Featherstone, was directly linked to the first CHS commitment, which relates to the relevance of aid in alleviating distress and suffering, upholding people’s rights to assistance and ensuring their dignity as human beings. In each round of surveys in Ethiopia, Lebanon and Mali, community perceptions of the relevance of humanitarian interventions to people’s needs improved. In Nepal, however, scores decreased over time.

‘While the changes were sometimes closely linked to a change in the internal or external context, it is noteworthy that NGOs were not always able to provide a specific explanation for the changes in perceptions,’ Featherstone writes. ‘Some felt that the act of engaging with communities and seeking to gauge their satisfaction may have led to better scores, or that monitoring and minor course corrections that strengthened the relevance of their programmes had a role to play, but NGOs often failed to identify specific programme modifications that would have driven the change in perceptions.

‘What is clear,’ Featherstone says, ‘is that while comparisons between NGOs in different countries can offer some interesting results, the most important unit of analysis is that of the individual agency and how it is perceived by communities that it is seeking to assist.’ The LLA project and the CV methodology shed important light on the strength of this relationship.

Interviews with NGO staff as part of the LLA project saw two clear benefits of the CV methodology compared to existing agency accountability mechanisms, according to Featherstone, namely ‘the prioritisation that it places on proactively engaging communities, and the perceived rigour of the process’. Proactively soliciting feedback about agency accountability instead of traditional reactive methods of getting information seemed to provide greater reassurance that issues were not only reported, but also stood a far better chance of being addressed.

The results of the LLA project demonstrate the ‘important need to maintain a dialogue with communities that permits changes in the operational context’, Featherstone says. ‘This is perhaps the most valuable contribution of LLA – and in so doing it provides a lesson in the importance of sustaining a conversation with communities affected by disaster.’

One concern was the time it took NGO staff to familiarise themselves with the project and its objectives, exacerbated by frequent staff turnover within the implementing NGOs on the ground over the course of the project. Constantly having to train new staff while carrying out a long-term initiative appears to have created productivity bottlenecks and increased costs.

Programmatic course corrections based on feedback varied from country to country and programme to programme. One of the surveyed communities in Lebanon, for example, revealed surprisingly limited knowledge of the assistance being provided by the local NGO, especially among women – who rarely left their homes. The NGO was apparently failing to adequately engage with this key client group. As a result, steps were taken to engage women more effectively so that they better understood the project’s goals and could participate more fully.

In Nepal, according to Featherstone’s comprehensive report, a second survey round carried out by an NGO providing sanitation services indicated that only 45% of those surveyed felt that the programme was meeting their families’ needs, while 67% identified potable water as their main need, with sanitation a secondary concern. ‘In discussion with their international NGO donor, it was agreed that the focus of the programme should shift to reflect the priorities of the community; the number of latrines was reduced and a water scheme was included in the programme. In addition, a livelihood programme was established in the villages which were most affected.’

In Ethiopia, community feedback raised concerns about camp security and the protection of children, resulting in action to refer these issues to police and government authorities. In Mali, feedback from affected people indicated a lack of information about complaints mechanisms and specific forms of programme assistance, prompting one of the NGOs to create a guidance note on accountability to facilitate better interactions with these communities. These relatively minor adjustments carried only modest cost implications.

Local NGOs, with fewer funding constraints, were more proactive than international donor-funded programmes, Featherstone reports, where ‘there was greater reticence to make changes or to engage with donor representatives about the need for change … it was considered by staff to be time-consuming and problematic to do so. It is also the case that agencies may limit their own flexibility by rigidly enforcing internal logical frameworks that act as a disincentive for making course corrections. In these circumstances, the penalties for making change are internal and bureaucratic, making them time-consuming rather than impossible.’

Such remediable shortcomings aside, the LLA project sparked the idea of going beyond individual projects and single organisations, combining the force of the CHS framework and the GTS methodology to place the views of affected communities at the centre of the design, implementation and monitoring of larger-scale humanitarian action. The countrywide project now under way in Chad (described elsewhere in this edition of Humanitarian Exchange) draws directly on lessons from the LLA project and applies them in a countrywide humanitarian response plan. It is still early days in Chad – and in Haiti and Bangladesh, where similar efforts are under way – but, like the LLA, it shows the promise of collaborative efforts in listening, learning and acting on feedback from affected communities.

Jeff Carmel is chief editor at Ground Truth Solutions, where Nick van Praag is executive director.