In 2019, this tool was developed in partnership with Ecorys, in consultation with a diverse range of social outcomes contracting practitioners, for the UK Government’s Life Chances Fund. It may be applied to other impact bonds but users should consider differences in context.
22 minute read
Have you investigated the problem and the wider system it fits in?
You have focused on a social problem and have investigated and identified the problems with current services, or the gaps in services.
You have explored reasons for the problems or gaps, and have distinguished between the social problem itself and the issues within the system or ecology of services that might have prevented it being tackled to date. For example, fragmented services, reactive spending and/or difficulty in changing underperforming or outdated ways of working innovating.
There is ambiguity about the social problem you wish to tackle, or there is no clear distinction between the social problem itself as opposed to the systemic issues which might have caused it or prevented it from being tackled. The problem you are trying to solve is not best addressed by a particular service stream and is difficult to measure robustly.
Have you gathered evidence around the problem and how to tackle it?
You have gathered learning to inform your analysis of the problem and options for tackling it, by reviewing your own learning and evaluation material, reviewing evidence from elsewhere, speaking to others who may have relevant experience, and gathering the opinion of current users of services.
You have not gathered adequate learning to fully analyse the problem or options for tackling it. Learning gathered/ available data does not indicate front-runner solutions or provides conflicting information.
Have you considered the costs and benefits of different solutions?
You have developed options for using different solutions to address the defined problem.
You have appraised the costs and benefits of different options, both in monetary and non-monetary terms.
This has enabled you to gain a clear understanding of the potential of Impact Bonds alongside other alternatives – for example, using outcome-based payment without social investment, or having a greater outcomes focus in current grants or service contracts.
Alternative options have not been properly considered, or their costs and benefits have not been fully understood.
Can you explain why an impact bond the best option?
Determine whether to use an impact bond through analysing if the benefits might outweigh the costs compared to other approaches. Keep in mind that benefits outweighing costs can apply to both monetary and non-monetary terms.
Consider whether anticipated benefits of impact bonds might apply: helping to promote collaboration between outcome payers and service providers, increasing the focus on prevention in spending, or triggering innovation in the way services are delivered.
There is no clear justification for the use of an Impact Bond over other alternatives.
Have you assessed viability by using the LOUD model?
You have determined whether an impact bond model might be viable by assessing your circumstances against the factors identified in the LOUD model: Leadership, definable Outcomes, shared Understanding, Data. In addition, the provider ecosystem and favourable regulations are key factors in enabling successful development of impact bonds.
You have a clear understanding of the resources required to develop the Impact Bond model, and where external support may be required, and you have budgeted for this.
It is not clear at the outset whether an impact bond model is viable in the circumstances.
Can you evidence why have you chosen your target cohort?
You have clearly communicated why you have chosen the target cohort and defined its characteristics (e.g. socioeconomic, demographic, incidence of the social issue). This is supported by an evidence base and stakeholder engagement.
You have poorly defined your cohort. There is no clear process of identification and there is ambiguity over the participants.
Facing ideological opposition or suspicion about SIBs from key decision-makers due to wariness of perceived 'privatisation' of public services.
Lack of understanding about SIBs, including limited skills and knowledge, and a lack of time and resource to gain the skills and knowledge needed. This makes it difficult for commissioners to communicate the rationale for developing a SIB to other stakeholders.
Concerns about the potential risks involved.
Some project leads who faced scepticism emphasised the longer-term benefits. For example, in one SIB, the project lead stressed that the SIB could be used to test an initiative, which, if successful, could be rolled out later as part of the core business.
Project leads found it helpful to build in sufficient time and resource to the process to ensure that they are knowledgeable about – and confident to – communicate the feasibility and appropriateness of the SIB to key decision makers. They found that many of the more technical aspects could be learnt as they went along, either by using tools or guidance, or by drawing on advisors’ expertise.
Several project leads highlighted that it was important to carefully consider the risks to their organisation in commissioning a SIB. If the potential reputational risk could be outweighed by the positive outcomes, then it was important that this was communicated to key decision makers.
Have you defined outcomes that have verifiable measures?
You have chosen a small number of clearly defined and specific outcomes that align with policy intent.
Each outcome has appropriate and measurable indicators that can be independently verified.
The outcomes have been poorly defined and they are not capable of being measured or verified
Striking the balance between easily measurable outcomes, and outcomes that are sufficiently personalised to the beneficiaries
It can be time consuming to agree on outcomes
Some projects leads overcame this issue by building in sufficient time at the beginning to allow for an iterative process of defining outcomes.
Project leads emphasised the importance of keeping outcomes simple (opting for a few key outcomes rather than many outcomes), but not at the expense of excluding important outcomes.
Have you set a price for outcomes and can justify it?
There is a clear articulation of price of outcomes and when they will be paid (payment milestones).
There is a strong justification based on an existing evidence base, which may be financial or historical data.
These outcome payments have been sense checked by the other stakeholders, namely providers/investors.
There is an unclear rationale for pricing of outcomes, or it is unclear how and when payments will be paid.
Having a lack of confidence and knowledge on pricing outcomes, due to not having the need to do it in the past.
Many successful project leads spent time at the beginning of the process to review how previous SIBs priced outcomes, exploring existing sources of evidence (see ‘Links to Resources' box), or by speaking to other commissioners to get the data.
Many project leads engaged an advisor who knew how to navigate existing data on pricing outcomes, or how other SIBs priced outcomes, based on their previous advisory work.
Does your financial model work in different scenarios?
The financial model has been developed and this shows the financial effect of different scenarios for both the volume of participants and their success at achieving outcomes and payment milestones.
All stakeholders (outcome payer, investor, provider) understand and support the financial model and understand the implications in terms of budgeting / financial planning / cash flow.
The financial model does not show the effect of different scenarios regarding volume of participants and/or success at achieving payment milestones.
The stakeholders either do not understand or do not support the financial model.
Knowing how to actually undertake financial modelling and having the skills in-house to do it.
Agreeing on a financial model that suits the needs of commissioners and investors.
One project lead undertook the financial modelling early on, as a way to calculate which outcomes would be viable. This helped to save time, because they started with draft outcomes definitions and values, and used the modelling to refine them (rather than defining and valuing outcomes first, and then finding out they were not financially viable).
Project leads emphasised the importance of making sure that they factored in a range of potential costs, including wider costs that might be borne by the local authority (such as performance management).
It was important for some project leads to ensure that there was a consensus among stakeholders on how rates of return were expressed.
Does your financial model show minimum and manimum investment required?
There is sufficient information to enable investors to derive a minimum and maximum investment requirement and when investment will need to be "drawn down".
Investors can be made clearly aware of the responsiblities and risks of backing the project.
Investment amount and timetable is not known, nor the risks and responsibilities created by the project.
Are you aware of risks, interdependencies and the needs of stakeholders? Have you planned for these?
A senior decision-maker has signed off the development process and an accountable person has been appointed.
A project planning group is in place to oversee development of the project.
You understand and have mitigated risks before and during the development process.
You are aware of interdependencies and stakeholder management needs of the project and will activate any actions if needed.
You have made a plan for the development process that included interdependencies and stakeholder management.
There is a lack of clear leadership of the project, or evidence of commitment from the required internal stakeholders within the outcome payer's organisation.
Roles and responsibilities are not clearly assigned.
Securing internal buy-in when there is misunderstanding or scepticism from colleagues about what a SIB is.
Churn within the commissioning organisation, making it difficult to maintain working relationships with different teams, such as finance or legal.
Some project leads engaged key decision makers early on, and spent time to explain what the SIB was and why it was beneficial to their organisation (over and above other approaches to commissioning)
In some SIBs, project leads developed a working group comprised of staff from different teams within their organisation (including finance, legal, procurement, as well as service leads) from the outset. In one area, the working group met on a regular basis to ensure that all teams were kept abreast of developments, so any emerging issues could be dealt with promptly.
Some stakeholders highlighted it was helpful to keep written documentation of the progress and decisions made throughout the SIB process because it meant that any new stakeholders could get acquainted with the SIB quickly.
Have you engaged the market to identify providers?
As an outcome payer, you have engaged with the market to identify new and existing providers. If needed, market development and testing has taken place, including capacity building of service providers and positive feedback on proposed terms.
There is no capability or capacity of the provider market (i.e. service providers, often charities or social enterprises), or this has not been tested or understood at large by the service providers.
You feel that you are not able to choose the best provider(s) as you have limited options.
Explaining the SIB mechanism to providers, who might not be familiar with it
Explaining the rationale for running the intervention through the SIB mechanism, as opposed to a more familiar mechanism, such as fee-for-service or grant
Some project leads engaged providers early on (after developing the technical aspects of the business case) to give them time to get a better understanding of what the SIB mechanism was, and how it might have affected them. It was also important for project leads to explain to the providers the reasons for running the intervention through a SIB.
One contracting authority engaged providers through holding ‘market testing’ days, where they invited prospective providers to learn about the SIB and discuss how best to set the procurement specifications. This worked well because it ensured all providers were on the same page, and it resulted in many interesting and high quality proposals.
Have the investors been engaged from the start?
Investors have been engaged in the development of the model since the start. The investor market and its representatives (e.g. intermediaries) have worked through the financial model and fully understand it.
Investors are only engaged in the later stages of the project.
There has been no testing of the expected target level of outcomes or price which influences the financial and impact proposition for investors.
There has been no testing of the investment required with investors or its representatives (e.g. intermediaries).
Which procurement approach are you taking?
The procurement approach has been decided (e.g. restricted process, open competitive process with dialogue, etc). The procurement considerations are appropriate, proportionate for both the funding mechanism and the service, and consider the implications on governance and service delivery.
There has been no consideration of suitability of the procurement approach, or of the implications of this on the wider stakeholder engagement required and the project as a whole.
Overcoming potential conflicts-of-interest in the design and commissioning process, particularly in provider-led SIBs.
Operating in often rigid structures of commissioning organisations’ standard procurement practices.
Some project leads involved the procurement team in the SIB development at an early stage, as part of a working group. This allowed procurement teams to advise about the process of provider market dialogue and collaboration.
For some project leads, it was important to review existing guidance and documentation (see ‘Links to Resources’ box) at the start of the process, so they could be aware of the different routes to procurement that could be taken.
Does your referral pathway avoid perverse incentives? Have you considered regulatory or statutory responsibilities?
A referral pathway has been developed (e.g. voluntary opt-in, proactive engagement or referral from another agency entity) which includes appropriate governance to avoid perverse incentives. Considerations have been given to the decision of the most adequate referral pathway.
The referral pathway has been developed in full knowledge and oversight of any regulatory or statutory responsibilities and with stakeholder engagement. A process is in place to determine eligibility based on the defined target cohort.
No clear process for accessing proposed project participants.
Developing referral pathways that prevent the cherry picking of people from whom outcomes might be easier to achieve.
In one area, the project lead established one route for all referrals to come through, with a specific screening tool built in to identify people with more complex needs. This allowed the contracting authority to have more oversight of the referrals that came in, to ensure that those with more complex needs were being supported through the SIB.
Are paitipcants involved and consulted in the deisgn and delivery of the intervention?
Either…
There is clarity on how to specify and access the desired interventions.
Or…
There is a strategy for co-designng the intervention with stakeholders.
Or…
In the case of a 'black box' intervention approach, a specific defined approach to ongoing contract and performance management is in place.
In all cases, participants are involved and consulted in the design and delivery of the intervention.
There has been no consideration of approach to intervention model and the effect of this on the project overall. It is unclear how the intervention model (or process for developing an intervention model) links to the proposed outcomes.
Choosing whether to have an evidence-based intervention (and obtaining appropriate licences) or having providers design the intervention.
Giving up control over the design of the intervention to providers.
One project lead found that holding a ‘market testing day’ with providers helped when it came to refining the specifications for the bid. They found that providers were more likely to propose a service that was in line with the contracting authority’s vision, as everyone had the same understanding of the rationale for the SIB.
Have you articulated how participatns will experience the service?
There is excellent articulation of how a participant will experience the service, including the role of the intervention(s).
The participant journey clearly integrates with existing services.
Unclear on what the participant journey looks like, or the integration with existing services have not been thought through.
Do you have a theory of change? What is your baseline and counterfactual?
A 'theory of change' to show how the project is expected to achieve improved outcomes has been articulated. There is a demonstrable understanding of the baseline and counterfactual and a clear plan in place for independent data verification and for evaluation of the project.
No consideration given to evaluation approach.
Are there any regulations you need to follow?
Any regulatory requirements relating to the services, such as Ofsted registration or safeguarding protocols, have been considered and planned for.
There has been little or no consideration of regulatory and legal requirements in the context of the proposed funding mechnism and service design.
Has the contract been signed?
An appropriate contract has been signed which satisfies all the areas of development: the business case, relationships and service design.
The contract does not satisfy all the areas of development and is therefore not viable.
Drafting an appropriate contract that aligns key stakeholders’ (commissioners, investors, and providers) interests so all parties are satisfied.
Some project leads engaged legal teams early on in the procurement process to advise and support on drafting the contract.
One project lead shared the draft contract with prospective providers during the procurement stage to help to speed up the contracting process.
Do staff know about the service and have they been trained?
There has been staff training and clear communication within the outcome payer organisation about the new service and how to utilise it.
External stakeholders, e.g., local service providers are aware of the project and how it integrates with other existing services.
Staff have not been trained about the project and there is poor communication within the organisation about how to use the service.
External stakeholders are unaware of how the project integrates with existing services and any promotional oppportunities have not been considered.
Gaining buy-in from other members of staff within the commissioning organisation like service managers and frontline staff.
Some project leads engaged with staff through workshops, or set up implementation groups (where frontline staff could advise on how a certain aspect of the SIB intervention could work). For example, in one SIB, the project lead conducted several workshops with staff to allow them to raise any questions or concerns they had about the SIB.
Do you know how you will reach the number of referrals needed and have enough resources to do so?
Referral pathways or methods of identifying and engaging participants are in place so that the agreed number of referrals will be reached.
There is adequate staffing, resources and skills in place across all parties to launch the service (including logistics like desk space and IT access).
Agreed referral pathways and/or engagement methods are not in place to ensure the number of participants will be reached. The required staffing resources or skills are not in place to launch and run the project.
How are you governing and managing the intervention?
The roles and responsibilities of the outcome payer, investor and service provider during the delivery are decided (plus intermediaries if relevant).
Appropriate governance and management structures are in place, and meetings scheduled.
Your governance structures allow for insight from ongoing service delivery to be analysed and understood and for ongoing service adaptation.
There is a lack of clarity about the roles and responsibilities of the outcome payer, investor and service provider.
The structures to enable project management and problem solving during delivery are not in place.
Have you got a system to report progress? Are you compliant with data sharing regulations?
Systems are in place to report on the project progress and achievement of outcomes, including methods to verify and validate data and to tackle data gaps or unavailable evidence.
Payment processes are in place (particularly where there are multiple outcome payers involved).
The project is compliant with data sharing regulations and required data sharing agreements and data consent processes are in place.
Major risks during delivery are identified, quantified and mitigated against.
Systems are not in place to measure progress towards outcomes, to report on the project, or to validate data.
The project is not compliant with elevant data protectin regulation and/or data sharing agreements are not adequate.
Risks have not been considered, or have been considered but not adequately mitigated against.
Having strong performance management processes in place.
Designing or utilising data collection tools that are fit-for-purpose.
Some project leads had discussions with the investors to identify the best approach to developing performance management systems. This was helpful because investors usually had experience doing this so they were well positioned to advise.
One project lead scoped out the service provider's ability to implement performance management systems. When they found it would not be sufficient, they agreed with the provider that they outsourced the performance management to ensure that it could be completed at the level of quality needed to meet the investor’s requirements.
How are you capturing ongoing learning? Does your evaluation integrate learning and share benefits?
There are processes and structures in place for input from stakeholders and ongoing learning.
An evaluation is set up which includes a process for integrating learning when the service ends, including realising savings or system-wide benefits where appropriate.
Ongoing learning is not facilitated or encouraged, including processes for iterating delivery approaches.
There is no plan to test the benefits of the service delivery, and/or no evaluation is to be conducted.
How are you managing the transition from launch to delivery? Who is accountable for contract performance?
There is a smooth transition from the team responsible for launching the project to those responsible for overseeing it (contract management)
There is a clear line of accountability for the performance of the contract within the outcome payer organisation
Team responsible for overseeing project are inadqueately briefed or do not understand specific requirements of outcome based contract
Accountablity for performance of the contract within the outcome payer organisation is unclear
Is data being used to inform decision making?
The provider is encouraged to make use of all available operational and evaluation data to inform decision-making during live running
The agreed reporting structure and reporting cycle is effective, and there is a clear feedback loop between reporting and decision-making, to ensure data is being used for learning and adaptation
Data is not being used to inform decision making during live running
The reporting cycle does not assist with decision-making or creates additional work with limited benefit
Are you tracking performance against the scenarios in the financial model?
Actual performance is tracked against the scenarios in the financial model, such that the financial effect of performance on providers and investors is continually understood
Outcomes data for payment claims is accurate and verified as agreed, and discussions take place to problem-solve if data availability is affecting the ability to claim payment
Actual performance relative to what was predicted at the start is unknown
Outcomes data is inaccurate, unverified or incomplete, and no action is taken to rectify this
Do performance management processes allow for changes to be made? Is there clear communication about issues?
The outcome payer participates in project oversight, and conducts performance management processes, according to the responsibilities agreed during set-up, and adjusts the approach as necessary according to the level of performance
There is a clear mechanism for ongoing communication between the outcome payer organisation and parties involved in delivery (provider and investor), including to remedy any performance issues
There is an open and collaborative approach to problem solving, together with the provider and (if appropriate) investor
Project oversight and performance management is lesser or greater than agreed at the start, or than what is necessary considering the level of performance of the contract
Communication between outcome payer, provider and (if appropriate) investor is difficult
Problem solving is not conducted collaboratively
Are there adequate resources? Will there be a review of the referral process?
The provider can continue to show adequate skills and resources for contract performance, and additional training for staff is agreed, including how to resource it, if required
The referral process is reviewed for effectiveness, and the client journey is well integrated with other services
There is ongoing communication to staff at all levels in the outcome payer organisation about the contract's aims and requirements from them
There is a skills deficit within the provider organisation with no mechanism for addressing it
Referral processes are not working as intended, and / or the service does not integrate with existing provision
Staff at any level of the organisation are unaware of the existence, or purpose, of the service
Are there flaws in the contract that need to be amended?
A clear and ongoing distinction is made between poor performance and flaws in contract design.
Flaws in contract design are addressed through negotiation, using an independent mediator if necessary. These might include:
Poor performance is mistaken for contract design flaws, or contract design flaws for poor performance.
Design flaws are not resolved even once identified and acknowledged
What added value has the impact bond had? What are the implications for the future?
The Impact Bond is achieving what was proposed in the business case, or there is good understanding of what additional value the contracting mechanism is adding (for example, is it helping to respond to fragmented services, reactive spending and/or difficulty innovating?), or a clear view is emerging that it is adding nothing
There is an ongoing understanding of the effect on a wider set of outcomes than those being paid for; interim evaluation findings are requested and considered
There is a growing understanding of the effect on the wider system and the wider application of learning, and the implications of this for future contracting decisions
Discussions begin on what follows the end of the contract, including the impact of this on users of the services: renew, repurpose, or re-think the approach
It is unclear whether or not the impact bond is providing additional value beyond an alternative method of contracting
Wider benefits or costs, and wider system effects, are not understood or acknowledged
No clear view is emerging on what to do after the contract completes: whether to renew, repurpose or re-think the approach.