Separating the ‘good’ failure from the ‘bad’: three success criteria for innovation

The value of innovation lies in its potential to improve humanitarian action. Innovation processes seek improvements by identifying and developing ideas for better tools, approaches and ways of thinking that will change the way humanitarian assistance is delivered. Innovators seek to do things better by exploring options for doing things differently.

The exploratory and uncertain nature of innovation means that some degree of ‘failure’ is inherent, as results will often differ from expectations. Innovation processes are dynamic. While they can eventually lead to improvements in humanitarian assistance, many false starts, unsuccessful pilots and revisions to original plans and prototypes occur along the way. It has been argued that, in order to increase innovation in the humanitarian sector, organisations and donors will need to become less risk averse and embrace ‘failing fast’ in order to support adaptation and improvement.

This poses a challenge to understanding what successful humanitarian innovation looks like. To address this lack of clarity, ALNAP has worked in partnership with ELRHA’s Humanitarian Innovation Fund (HIF) to carry out 11 in-depth case studies on innovation projects funded by the HIF. The aim of this research is to understand the factors that contribute to successful innovation in humanitarian contexts. This is the first empirical research in the humanitarian system that examines successful innovation processes and the factors that contribute to them.

Why is this important?

There are three reasons why we should care about defining success criteria for humanitarian innovation. First, innovating teams and organisations need a way to communicate to donors and other external actors that they are making progress even when faced with an unsuccessful pilot or initial prototype. To do this, we need ways of distinguishing ‘good’ fails from ‘bad’ fails, which requires greater attention to the ways in which innovation processes generate evidence and learning from their pilots.

We also need to know what successful innovation looks like so that it can be better supported with guidance tools on good practice. Recommendations for how to innovate for humanitarian purposes must be grounded in solid evidence that links particular practices to successful innovation. We can only do this if we have a clear understanding of what successful innovation looks like.

Third, understanding what constitutes successful innovation can help us look more carefully at how we measure success in humanitarian action more generally. Innovation is concerned with generating improvements. In order to determine whether an improvement has been offered, we need quality baseline data and/or consistent standards that can be used to make comparisons across different approaches, tools and interventions. Such practices are significantly lagging behind where they should be in the humanitarian system to date. Demanding high levels of evidence from innovations while the evidence on the effectiveness of current interventions remains weak creates the possibility that we might hold innovations to a higher standard than existing practice. This might constitute an unreasonable level of risk aversion that sacrifices great potential gains in improved assistance. The demand to demonstrate the success of innovation processes requires us to also look harder at how we measure the performance of current practices and interventions.

Defining successful humanitarian innovation: three core criteria

We can identify the criteria for successful innovation by reflecting on the value that an innovation process offers across a range of scenarios:

• The ‘ideal’ scenario, in which innovation is fully successful and has causally contributed to improvements in humanitarian action.
• The ‘missing middle’ scenario, in which an innovation has developed an effective idea for improving humanitarian action but may not have been adopted by many humanitarian organisations.
• The ‘good fail’ scenario, in which the original idea turned out to be ineffective or unfeasible, but lessons are generated that can support future successful innovations.

These scenarios are not mutually exclusive. In fact, the pathway to many successful innovations includes individual cycles that end in a ‘good fail’ or ‘missing middle’ scenario. For some innovations with long incubation periods behind them, each of the three scenarios has occurred at some point in the trajectory of the overarching innovation process.

The ‘ideal’ scenario: adoption

Beginning with the ideal case, the best possible outcome of an innovation process is the wide adoption of an innovation that then contributes to better performance in humanitarian action. Adoption is therefore a criterion of success for innovation processes: it is the most desirable success criterion and the most difficult to achieve.

The ‘missing middle’ scenario: improved solution

In the humanitarian literature, it is often implied that adoption is the only criterion of success, and that successful innovation equates to products or processes being taken to ‘scale’. However, this fails to acknowledge the contributions of innovation processes that struggle with what has been called the ‘missing middle’ of innovation. Dan McClure and Ian Gray, ‘Scaling: Innovation’s Missing Middle’, paper delivered at the Humanitarian Innovation Project conference, 19 July 2014. The missing middle refers to the gap between developing a product or process that offers an improvement over prior practice and achieving wider uptake of that product or process within the sector. Innovating teams can fall into this gap if there are adverse incentives in the humanitarian system that block uptake, even if the product or process offers tangible improvements for humanitarian action. Cash-based programming is often cited as an example of this kind of innovation. So-called ‘product’ innovations, such as pieces of software or water and sanitation kits, can also offer improvement, yet fail to reach full adoption. This is due to the ‘wasn’t built here’ syndrome, where humanitarian agencies prefer to create their own version of an innovation rather than adopting one built by another organisation.

In these cases, HIF grantees used a variety of methods to overcome the missing middle, from generating strong evidence around their innovation and using this as a basis for advocacy to targeting specific actors who could help overcome the institutional barriers to adoption. For some innovating teams, attempting to address all political and institutional barriers may not be a reasonable expectation or a good use of resources. In these cases, senior leaders within an organisation o en must buy into the innovation and use their influence to help its wider adoption. The point, however, is that innovating teams can do everything correctly, not achieve adoption and still claim some success for their innovation process if they have generated improvement.

Assessments of innovation processes must acknowledge the difference between cases where scale is not achieved but the innovation is a good one, and cases where scale does not occur because the prototype does not work or because the innovating team did not manage the process effectively. If an innovation process produces a good innovation it may still qualify as ‘successful’, in so far as it has yielded a viable improvement over current practices. Developing an improved solution for humanitarian action is therefore a second success criterion for humanitarian innovation.

The ‘good fail’ scenario: consolidated learning and evidence

In some cases, it may turn out that an innovation does not offer a viable improvement over current practices. The original idea for the innovation may have turned out to be unworkable in a way that could not have been expected at the outset of the innovation process. In these cases, innovating teams can still make an important contribution by sharing consolidated learning and evidence from their innovation process, in order to assist others who may try to build on their attempts or work on a similar problem in the future. These cases might still be considered ‘successful’, in so far as they contribute to the body of knowledge that is necessary for the humanitarian system to make progress. Therefore, consolidated learning and evidence is a third criterion for success that an innovation process might meet.

There are several good examples of consolidated learning and evidence being used as a crucial building block for further work towards an eventually effective and successful innovation. In 2005, Save the Children UK and the Emergency Nutrition Network set out to understand the effectiveness of supplementary feeding programmes (SFPs), a common intervention for acute malnutrition. Carlos Navarro-Colorado, Frances Mason and Jeremy Shoham, Measuring the Effectiveness of Supplementary Feeding Programmes in Emergencies, HPN Network Paper 63, October 2008.  Their initial research, published in an HPN Network Paper, found that there were significant problems with the quality of monitoring data collected on supplementary feeding programmes, as well as highly inconsistent use of reporting categories and measures. As a result, SCUK and ENN led a consortium to develop a set of standard reporting requirements, initially for moderate acute malnutrition and later also severe acute malnutrition. SCUK also led the development of new software to enable the easy and consistent collection of data to monitor the performance of SFPs.

The process featured many stops and starts and restarts, as different types of software were trialled, and as the project changed to fit broader changes in malnutrition programming, including a shift towards Community-based Management of Acute Malnutrition (CMAM). In each iteration of the project, including unsuccessful pilots of an Access-based version of the software, lessons learned fed into the next cycle (though not without difficulty, as many of these lessons were generated according to funding deadlines rather than as a natural part of the innovation process). SCUK ended up carrying forward the project on its own, resulting in the recently launched CMAM Report, which is now being used by nine agencies in 20 different countries. The consolidation of learning and evidence in this project not only assisted SCUK in eventually developing and diffusing an effective innovation, but also contributed more widely to the nutrition sector’s understanding of the effectiveness of CMAM programming.

Three success criteria

Over the past year, ALNAP has been using three criteria for successful innovation defined from the above scenarios in order to explore what factors and practices contribute to the successful management of humanitarian innovation. Working with 11 HIF grantees, ALNAP has explored the approaches and tools used by grantees and the extent to which they have helped to achieve the following three success criteria:

• Consolidated learning and evidence: new knowledge is generated or an enhanced evidence base around the area the innovation is intended to address, or around the performance of the innovation itself.
• Improved solution: the innovation offers a measurable, comparative improvement in effectiveness, quality or efficiency over current approaches to the problem addressed by the innovation.
• Adoption: the innovation is taken to scale and used by others to improve humanitarian performance.

Findings on the factors that contribute to the achievement of these success criteria are discussed in the final synthesis report that ALNAP and ELHRA are publishing in April 2016.

Other considerations

While the above criteria are the main way in which we understand innovation to be effective and successful, other factors can be important in our assessments of innovation:

Involvement and respect of affected people. Demonstrating how the rights and interests of affected people are respected in an innovation ought to be a minimum standard for all innovation processes. Too often, the message that humanitarian agencies should be less risk averse can overshadow the fact that increased risks are easily passed onto affected communities. Humanitarian organisations must take specific measures to ensure that any increased risk in terms of cost-effectiveness remains confined to the innovating organisation, rather than the affected community. As found in the ALNAP-HIF case studies, using a staged approach to piloting, whereby pilots are first undertaken in non-emergency contexts with clear protections and benefits in place for participating communities, is one way to deal with this.

Efficiency. Innovation processes can often appear weak on efficiency, particularly when they involve the development of new technologies or tools. There are, however, clear best practices that organisations can use to improve the timeliness and thus efficiency of their innovation process. For example, having a clear division of tasks and responsibilities across the innovating team and partners was supportive of an efficient innovation process. Also important are well- planned pilots that include defined times for collecting and responding to feedback, and that are implemented in a way that is complementary to the standard operating procedures, organisational structures or practices of pilot participants.

Unique impact. The individual unique impact of any innovation is often a function of its novelty, which in turn is shaped by how much the sector changes as the innovation process takes place. When a particular issue, such as cash-based assistance or menstrual hygiene, is largely ignored by the humanitarian
system, innovations that seek to develop solutions in these areas can carry a high degree of risk, but also a unique impact on the system around them. As other humanitarian actors become more sensitised and active in these issues, innovations may not be able to offer a unique impact, but can still contribute to the sense of a ‘groundswell’ of activity that can serve as a tipping- point for the wider adoption of effective tools and approaches.

Looking ahead

Further attention to and research into the performance of innovation processes is needed if innovation is to deliver on its promise of improving humanitarian action. Given the enormous challenges in delivering timely, relevant and principled humanitarian assistance, fulfilling this promise is imperative for both the humanitarian crises of today and of tomorrow.

Alice Obrecht is a Research Fellow with ALNAP.