This blog is written by Lorcan Clarke, Research Officer at the LSE and former Research Associate at the GO Lab and Clare FitzGerald, Research Fellow at the GO Lab.
Impact bond projects are being implemented the world over, but it’s difficult to know for certain whether they are living up to expectations. Organisations in the business of evaluating and using impact bonds face several challenges. First, they must assess the credibility of claims of positive ‘impact’ and in doing so, decide what the necessary amount of ‘robust’ evidence is to make conclusions, including whether impact bonds are cost effective. This blog introduces our investigation into the use of economic evaluation tools to address these challenges.
For some, a key premise of the potential for impact bonds is the improvement of public services through incentives to provide quality information on service delivery and impact. In the short term, this allows for course correction during project implementation and in the long term, improves commissioning decisions based on what ‘works’. For this to prove true - for impact bonds to catalyse better use of information in the delivery of public services - we need high quality, widely accessible evidence on their delivery, impacts, and costs.While scholars have explored how impact bonds have worked to support innovation in a broad sense, there appears to be a gap in the literature for work that studies the amount and quality of evidence impact bonds have generated, at least in the public domain.
Sensing this gap, particularly around cost effectiveness, we began a systematic review of economic evaluations related to impact bonds in November 2018. We focused on impact bonds targeting health outcomes, to draw upon available guidance estimating the economic value of health-related policies and interventions.
Here’s what we’ve found so far:
Our search involved screening over 450 studies obtained from Google Scholar and databases of articles published in academic journals. We also hand searched online databases held by involved parties, a 2018 list of all the impact bonds targeting the social determinants of health, and PIRU’s 2016 systematic review on impact bonds. To be included in our review, the evaluation had to involve a project where: 1) upfront financing is tied to a results-based contract, 2) separate parties of investors, providers, and outcome funders are involved (or proposed), and 3) a payment is directly linked to improved health outcomes. The evaluation had to include a quantitative analysis of both the intervention and the use of an impact bond. This meant we excluded studies which only described the calculation of outcome payments.
After screening on these criteria, we found 11 evaluations of interventions targeting health outcomes funded using impact bonds, just over a handful each for us to review. The interventions cover 5 countries: The United Kingdom, the United States, Portugal, Chad and the Czech Republic. None of the evaluations are linked to a launched project funded by an impact bond. though some examine similar interventions in different contexts. One example of this is an analysis of using a pay for success approach for childhood asthma interventions among those in the Medicaid population in Detroit, Michigan, which aligns with a project in Baltimore, Maryland (United States). “Pay for success” is the term used in the United States for financing structures which are similar to impact bonds. We identified several assessments that include health and health-related outcomes in the rationale, but not the outcome metrics, for a project and excluded them on this basis.
Of the 11 evaluations we identified, several were conducted by organisations directly involved in the proposal or implementation of impact bonds. This includes social impact bond intermediaries and research units working on assessing and promoting social impact bonds for implementation. Of publications reporting funding sources, only one was funded independently - the rest were from actors directly involved in impact bond projects.
There is precedent for routinely conducting and publishing this kind of research. Although not perfectly analogous, biomedical and health services research annually leads to publication of thousands of evaluations that look at the value for money of medicines and healthcare services. A variety of organisations are behind the funding and publication of these evaluations, from academia to public organisations, consultancy firms and private organisations providing the good or service being evaluated. Independent research and commentary evaluates the methods and results of this evidence - often via systematic reviews and meta-analyses - and address potential publication bias. Ensuring that evaluations are in the public domain and independent research is supported is critical to any advancement of impact bonds or any kind of results-based financing.
Evaluation approaches across our included studies varied. Even on basic terminology, studies looked at value for money through analysis of cost-estimates, comparative benefit-cost, social return on investment, and feasibility. None of the studies reference, or outline, standard criteria for estimating the value of using an impact bond, or similar mechanism, to finance a project. This highlights the need for to create and promote adoption of standard methods, including ones already made available - such as those published by the Urban Institute and Government Outcomes Lab.
Standardised approaches allow policymakers to make transparent decisions about funding public services, and citizens and taxpayers to be aware of what information was used to justify those decisions. Standard evaluation practices allow stakeholders - outcome payers, service providers, and upfront investors alike - to sit in the same room and reliably have the same understanding of a situation. In the United Kingdom, the National Institute for Health and Care Excellence (NICE) has standard measures for evaluating the quality and impact of healthcare interventions. Much effort has been put into establishing comparability in the evidence published on clinical trials, cost-effectiveness analyses and systematic reviews. There is opportunity to create and implement similar guidance for impact bonds, and support debates over value for money that complement those on theoretical merits and project implementation.
Evaluation is key to understand if impact bonds are a useful structure for financing and implementing public services, even in very specific circumstances. Rigorous and independent analysis is just as critical to understanding complex contracts as it is for the complex molecules that make up medicines. Misunderstanding these complexities can create confusion and missed opportunities to improve the use of public, charitable, and social investment funds. Impact bonds and pay for success contracts are not alone in facing evaluation challenges - “green” and “sustainable” investing approaches face similar attribution issues - but we must start somewhere, and soon