This guide will walk you through setting and measuring outcomes. While it covers what you need to consider if you are developing an outcome based contract or social impact bond, the content is relevant for anyone who is trying to work out how to set and measure outcomes for any purpose. It will explain what an outcomes framework is, how to identify and measure outcomes with examples, and explain key concepts
This is the second edition of this guide. We would like to thank Neil Stanworth from ATQ Consultants who wrote the first edition of this guide. He shared considerable expertise and experience with us and we were able to build on this for our updated version.
By definition, setting the right outcomes, and measuring them effectively, is one of the most important aspects of an outcomes-based contract. The achievement of outcomes will determine how much is paid by the commissioners to the providers or investors. Also, the definition of outcomes directly affects whether commissioners achieve value for money from the contract.
This guidance is about setting and measuring outcomes. If you are putting together an outcomes-based contract, we would advise you to also read the Payment Mechanism guide (to be published December 2018). And regardless of what sort of contract you are letting, we would advise you to read An Introduction to Evaluation. As outcomes inform payments to be made, and the impact of the outcomes need to be measured and evaluated, these three guides are inextricably linked. We will refer to these guides and other resources throughout.
Furthermore, our projects database shows all the SIBs that have been launched and identifies their outcomes. You can search different policy areas so you will be able to find a SIB that is similar to one that you are developing.
For more in depth information you can look at our case studies. These focus on a smaller number of SIBs in greater detail and are a great resource to look at to see what others have done.
2 mins read
The overarching point is that one very good reason to set and measure outcomes is formative. You will use the information to indicate whether a project is moving closer to its ultimate objective or not. You will make decision based on the information. For example, you might offer some advice to a provider, or offer your team a certain type of training.
Setting and measuring outcomes gives you more information to ensure you are making the right decisions that will move you towards your ultimate objective. It's not enough information on its own as you will want qualitative information as well, to add colour to the numbers. A good combination is best.
Measuring outcomes is a very important part of the process of evaluation, which we cover in a separate guide, see An Introduction to Evaluation. Some evaluations can be formative as they help with learning how to do better. However, this point refers to the summative evaluation which comes at the end and gives you an idea of how effective it was.
An impact evaluation is the most robust and uses quantitative methods (statistics) to show whether a project has or hasn't produced better outcomes than what might have happened anyway. Qualitative evaluation (discussions, interviews, stories) is also needed to understand why something worked or didn't. Together, this information can help you and others to decide whether to continue a project, change it, or stop it altogether.
Outcome based contracts have seen increasing use in the UK in recent years, particularly through social impact bonds. These are where partial or full payment is made between a government agency and an external provider (or sometimes, between different levels of government) based on a set of pre-agreed outcomes.
The idea is that by paying for outcomes, rather than specifying particular activities (which is a more typical method of contracting), the commissioner (or 'payer') gives the provider more flexibility to deliver the service in a way that will work best, learning as they go along.
Proponents also argue it gives providers a greater incentive for success, and reduces the financial risk to government if a project fails. To pay for outcomes, you need to be extra thoughtful about how you set and measure them. After reading this guide, you may wish to read the Payment Mechanism guide.
Once you have an understanding as to why you are measuring outcomes, you can then begin developing an outcomes framework.
3 mins read
A robust outcomes framework sets the groundwork for your SIB project. It needs to define the following:
When developing this framework, it is useful to refer to the contract triangle. This resource was developed to set out the key components of developing an outcomes contract. It explains that an outcomes contract needs to be robust on three things: the cohort of beneficiaries that your SIB project will support, the outcomes that the SIB aims to achieve, and the price that will be paid for the outcomes. This technical guide will focus on the second point in order to support you to come up with outcomes that align to the policy objectives.
Both measures and targets may relate to what is achieved by and for an individual within the cohort, or to performance across the cohort as a whole.
It is helpful to think about outcomes and measures separately when considering the framework as a whole. It may be possible to define outcomes that fully relate to the policy objective, but measures may not be comprehensive in capturing all facets of an outcome. The fact that an outcome is difficult to measure does not make the outcome invalid, but it may mean it is not appropriate to attach a payment to it (as you might in an outcome based contract).
It is also advisable to have a fairly settled view of the outcomes and measures to be applied to a contract before starting to assign specific values to them as metrics or triggers.
7 mins read
With an understanding of the key concepts you will need to start identifying the right outcomes for your project. The chapter will start with a checklist for identifying the right outcomes. It will them go into more detail about the different types of outcomes there are so you can consider what might be appropriate for your project.
It is important to note that there are no hard and fast rules about what constitutes the “right” set of outcomes and measures for a particular project or contract. Project teams should expect that the process of developing outcomes will be both iterative and progressive.
Iterative– It is likely that developing an outcomes framework many outcomes will be discarded and there is an element of trial and error. For example, if there was challenges to do with data collection, the outcome may not work.
Progressive – In the early stages outcomes don’t need to specified and defined in great detail, but as work progresses they will need a closer definition. Ultimately, they will need to be clearly and unambiguously defined in the contract.
Different stakeholders may have different perspectives on what constitutes a robust and viable outcome measure and it is crucial to negotiate as you go along. In most cases, the task of setting outcomes is likely to proceed incrementally as you develop the business case. It will be informed and influenced by other decisions, particularly designing the payment mechanism for your outcomes (if designing an outcomes contract).
When you are selecting the correct outcomes you will need to pin down the primary outcome and the other important secondary outcomes.
primary outcome – this is the single most important outcome from the contract, and the one that the commissioner most wants to see positively impacted. It is likely to reflect the most important outcome for the commissioner in policy terms and/or delivers the greater financial benefit to the commissioner.
secondary outcome - these are the other important outcomes that the commissioner wishes to see improved. They may capture a different dimension of the programme intent and help to reinforce and sustain the primary outcome. They may capture stakeholder interests that are not reflected in the primary outcome and offer further benefits to the commissioners. They may also counterbalance perverse incentives.
Limiting and classifying the number of outcomes is simpler to manage, as having multiple outcomes make it harder to predict when they will occur. In an outcome based contract, including multiple payment outcomes makes it harder for commissioners to forecast expenditure, and harder for providers (and investors) to predict income and cashflow.
At this point it will be useful to refer to the payment mechanism guide to understand how outcome measures are linked to payment.
The main advantages are:
The main disadvantages are:
A binary outcome is a type of hard outcome that has only two states, either an outcome is achieved or it is not. They are used where it is deemed unacceptable for the public sector to pay for outcomes that include negative events. For example, commissioners may run a project to reduce re-offending, and may wish to track its reduction – but this is not binary, and offences will still have been committed. Commissioners should consider the political impact of the measurement and whether a binary outcome may be most appropriate.
Soft outcomes depend on measurement which is more subjective, e.g. an individual’s self-assessment. They are nearly always measured at the level of the individual or family rather than across the cohort. Soft outcomes are usually set and measured by referring to proprietary measurement methods, such as Outcomes Star family of tools or Strengths and Difficulties Questionnaire (SDQ).
The main advantages are:
The main disadvantages are:
Below are two examples of setting outcomes and how they divide into hard and soft. Both these examples are from our case studies that provide in depth look into many aspects of the projects in question.
Hard outcomes – Increase in school attendance, improved attainment in English and maths
Soft outcomes – improved social and emotional skills, peer relationships and conduct. These are measured through an improvement in SDQ score (strengths and difficulties questionnaire)
Hard outcomes – reduced hospital admissions, reduced use of outpatient and A&E services
Soft outcomes – improved wellbeing, measured through Triangle Consulting’s Wellbeing Star
Proxy outcomes have been defined by The Centre for Government (US) as an “indirect measure of the desired outcome which is itself strongly correlated to that outcome used when direct measures of the outcome are unobservable and/or unavailable”.
As the name suggests, a proxy outcome acts in place of the actual outcome you are interested in. It is something that tends to go hand-in-hand with the desired outcome – if one is achieved, the other is achieved and vice versa. But beware - correlation does not mean causation so you may end up paying for outcomes which have little to no effect on the actual policy intent of the outcomes-based contract.
Many soft outcomes and measures are used as a form of proxy on the basis that they will indicate achievement of another outcome that is difficult to measure.
There are many examples of proxy measures and how they can help when outcomes are not easily measured, but there are challenges too.
Lead or progression outcomes show movement towards a later outcome.This may be necessary if the primary outcome take a long time to measure and commissioners want reassurance in the meantime that the outcome is likely to be achieved.
For example, for children with highly complex needs in foster care, a lead measure might be better attendance at school, where the evidence indicates this increases the likelihood of the child maintaining a stable foster care outcome. There is also value and impact for the individual and the service from that outcome in its own right. The boxes below offer more examples of how progression outcomes can lead to other greater outcomes.
Lead or progression measures are also used when there is a low likelihood of achieving the primary outcome for the entire cohort, because people have different levels of need. Therefore, some people many require a different starting point and the acknowledgement that are meeting some intermediary outcomes along the way, as indicated by lead or progression measures.
In an outcome based contract, where payment is placed against the achievement of outcomes, payment against these lead or progression measures may help to mitigate the risk of “parking”, whereby the provider chooses not to work with certain individuals who are too far from achieving the outcome. Doing this can considerably complicate the design of the payment mechanism, though.
The main disadvantage of lead or progression measures is that providers may not invest in moving the cohort towards the primary outcomes if the payment terms provide significant reward at this earlier stage. The level of payment attached to such outcomes, and the number that can be claimed across a cohort, needs to be set with care.
3 mins read
Commissioners should carefully consider how an outcome will be measured and how the data needed to support each measure will be collected, reviewed and reported. This section will set out a checklist for measuring outcomes and then go into greater detail about the data collection method, and whether you should measure outcomes across a cohort or for an individual.
As measuring the outcomes is about knowing what the impact of the project is, it is important to consider evaluation at this point. Refer to our technical guide to understand what you need to consider for your evaluation.
In terms of measuring outcomes, the method of data collection is crucial. You will need to find out whether the required data is already collected for other purposes. Assumptions should not be made about the availability of data from other parties or the ability of those parties to collect data on your behalf. If certain data is needed to measure an outcome, its availability, the legality and practicality of collection should always be confirmed.
The table below sets out the types of data and the pros and cons of each approach.
The success of the contract in achieving better outcomes (and triggering payment where appropriate) can be measured in two different ways: the individual (or family unit), and the cohort.
If you are designing an outcome based contract, choosing individual or cohort measures will impact on the payment structure. The majority of outcome-based contracts have used a rate card linked to individual outcomes because this is cheaper to manage, and can be simpler than a cohort level measurement approach. To understand more about this, look at the payment mechanism guide
The commissioner needs to consider which is most appropriate for the characteristics of the cohort and this is highlighted below.
Works best with a homogenous cohort. When all members of the cohort experience the same or similar adverse outcomes the outcome target can be set at an individual level. E.g. avoiding entry to care for a cohort who are all at risk of entering care.
A comparison group is not required. As the individual is assessed and this directly relates to the outcomes there is no need to set up a comparison group. Changed or improved outcomes can be specified on a rate card that shows the payment amounts per individual when an outcome is achieved.
Does not allow for what would have happened anyway (the ‘deadweight’). If there is no comparison group, the outcomes achieved are likely to include some that would have happened anyway, without the intervention (either because of other positive things happening in that individual’s life, or due to natural statistical variation). This can only be avoided if the commissioner has a reliable way to estimate what would have happened anyway (the deadweight), or is certain that the intervention is being applied only to people who would not otherwise achieve an improved outcome. In an outcome based contract, if this is not done, there is significant risk that the provider/investor will be rewarded for achieving too little impact., so the payment level should be adjusted downwards accordingly.
Works best if adverse outcomes vary considerably. It is more difficult to set standard measures of success at the individual level if the cohort is made up of different individuals. E.g. the frequency and severity of offending can vary widely and a rate card could not be used.
Usually requires a comparison group – this is needed so the outcomes achieved through the contracted intervention can be compared against a group up did not receive a service. This is known as a counterfactual and is there to check that the outcome would not have happened without the intervention. In an outcome based contract, this can be used to assess whether payments should be made.
Does not require separate calculation of what would have happened anyway (“deadweight”)– any deadweight will be reflected in the measurement against the comparator group, and only performance over and above the comparator group will be rewarded.
The One Service SIB in Peterborough used a cohort measurement. You can find out more about the methodology used to measure the impact of the SIB project here.
The Innovation Fund SIB by DWP used an individual measurement. This was done through a rate card whereby the price was set for each outcome. You can read more about rate cards in the payment mechanism guide.
6 mins read
Commissioners will need to develop outcome metrics so they can measure how success will be judged and when payments will be made. If outcomes have been well written, setting appropriate metrics should be relatively straightforward. The metrics should align to the social objectives of the contract and to the anticipated financial benefits.
This chapter will set out a checklist for setting outcome metrics, before going into greater detail about the considerations that need to be made. Note that this should not be viewed in isolation as when outcomes are set for an outcome based contract, there will need to be decisions made about payment (see payment mechanism guide). Also, as measuring outcomes are about understanding the impact of the SIB project, you will need to think about how you will evaluate the project (see An Introduction to Evaluation)
In a contract that incentivises skills attainment by young people, there may be a metric (and payment) relating to:
In a contract that rewards the achievement of employment there may be similar metrics relating to the person:
As in many aspects of contracting, it is important that the decisions made regarding the outcomes framework are tested with potential service providers and, where appropriate, investors.
This applies especially to outcome based contract where outcomes form all or part of the payment, where it could be argued that ensuring the outcomes framework is acceptable to all parties is the single most important consideration. If the commissioner is prepared to pay for the achievement of an outcome, and providers and investors are comfortable to have success measured by the achievement of the same outcome, a successful contract is likely. Conversely, if one or more parties are not happy with the proposed outcome measure or the performance level required to achieve payment, it will be very difficult to conclude a contract successfully.
Commissioners may wish to consult other parties while designing the service, and should do so prior to any competitive process. This will judge how acceptable the proposed framework is. It will also be useful to benefit from providers’ (and investors') previous experience of similar contracts and what outcome measures and metrics were used then.
There are two main forms of perverse incentives, these will be discussed below with examples and ways to mitigate them.
Sometimes providers, investors or intermediaries may select beneficiaries that are more likely to achieve the expected outcomes and leave outside the cohort the most challenging cases. This is known as ‘cherry picking’.
An example of this is when an outcome is chosen that seeks to achieve an absence or reduction of referrals to a statutory agency. The perverse incentive is to simply not refer people even if they fit the criteria for statutory support. This will make it look like there has been a reduction even though it is not in the best interests of the beneficiaries.
To mitigate this there should be a collective decision-making process involving both the commissioner and service provider, or a neutral referral party/mechanism. This provides the opportunity for all parties to make and agree decisions that are in the best interests of the beneficiary and avoid ‘cherry picking’.
In some instances, providers may choose particular people from the cohort who are already nearer to the desired outcome and therefore are easier to support – this is known as ‘creaming’. The opposite of this can also happen whereby providers neglect people who are less likely to achieve positive outcomes – this is known as ‘parking’.
This may occur when an outcome target does not consider the varying levels across the cohort. For example, an outcome measure may be ‘an employment support programme requires 50 number of people to be in employment in 12 months’. The cohort may be made up of people who have very different needs. More challenging cases such as those with complex needs and who are longer term unemployed may be ‘parked’ and those who are likely to find it easier to get a job may be supported more. To mitigate this the outcome metric could be structured so that the start point of the individual is recognised and they are rewarded for the progress made. (See lead/progression outcomes.) This will mean that variation in the cohort is accounted for.
It may also occur when there is a simplistic binary outcome that results in no support being given after a certain milestone – a ‘cliff edge’. For example, a homelessness project may seek to settle 20 many homeless people in 12 months. If people fail to achieve this outcome they are no longer of value to the provider and are ‘parked’. To mitigate this, the target should reward success at small and regular intervals, or offer bonus payments. The amount of homeless people settled could be measured at 12 months, 18 months and 24 months and bonus payments could be set to encourage people to settle in accommodation.
You will need to consider how you will mitigate against perverse incentives so that the outcomes are in the best interests of the beneficiaries. If you refer to the contract triangle and make sure you do the three key points well, you should avoid perverse incentives.
Whilst these external factors may not be under control, it is important to determine the potential impact they could have when designing the service. It is wise to take into account the likely influencing factors in the metrics, they could be both positive or negative. It is also wise to have agreements with other practitioners so they all operate in a way that enables the outcomes to be achieved.
In some circumstances, commissioners may accept obligations under the contract to manage the impact of these factors, but this is often not the case. It may be preferable to include the monitoring and reporting of these impacts as part of the contract management process, with the expectation that the commissioner will take steps to mitigate or avoid where possible.
If you look at our case studies you will find examples of outcomes measures and targets that have been designed in a robust way.
2 mins read
As there is no ‘correct’ or ‘set’ way to create outcomes, measures and targets you may be questioning what good looks like. Having understood the considerations that have been outlined in this guide you are equipped to write a robust outcomes framework.
You can test your outcomes using the simple decision tree below. Once you have identified the outcomes and their appropriate measures you can check the outcomes definition section of the SIB readiness framework to see what best practice will look like.
West London Zone designed a robust outcomes framework and this section will paraphrase from their website.
As part of the Collective Impact Bond Steering Group process, WLZ worked with key stakeholders in the area to analyse some of the challenges that children and young people growing up here face. The goal was to understand where to focus efforts, and what outcomes we should be aiming to achieve over the long-term. Many areas were identified at each age stage.
WLZ took these and distilled them down into an ‘outcomes framework’, which intends to show the progress a young person is making towards flourishing in all areas of their life. WLZ has attempted to find a way to reliably track the ‘whole child’. The WLZ whole child outcomes framework is enables all those working with a young person to understand and track their progresstowards flourishing on an ongoing basis.
The framework covers a total of eight ‘outcome areas’ across the three domains of wellbeing, learning and character. All eight are measured and monitored for school-age children, but a lower number are measured and monitored in very young children and young adults as they are not all relevant. Each of the ‘outcome areas’ has one or two measures which enable the monitoring to take place
Data on each of these metrics will be collected annually through the WLZ Data Collection Survey as well as through secondary data already collected by schools and, where appropriate, other statutory sources. Some will be collected more regularly through delivery organisations.
You can view more examples of outcomes frameworks on the case studies as part of our website.