Noise Solution Social Return On Investment Study 2019

Social Return on Investment Report Noise Solution July 2019

Prepared for: Simon Glenister and Damien Ribbans, Noise Solution Prepared by: Naomi van Hest, Harriet Cant, Natalie Hearmon, Lucy Eddowes, Costello Medical

Noise Solution – Social Return on Investment Report July 2019

Contents Introduction ............................................................................................................................................ 4 Noise Solution ...................................................................................................................................... 4 The Importance of Wellbeing ................................................................................................................ 4 Social Return on Investment (SROI) ...................................................................................................... 4 Objectives for the SROI ........................................................................................................................ 5 Methodology ........................................................................................................................................... 6 Scope and Stakeholders........................................................................................................................ 7 Time Frame of SROI ......................................................................................................................... 7 Stakeholders Considered ................................................................................................................... 8 Investment .......................................................................................................................................... 9 Outcomes and Evidence ....................................................................................................................... 9 Impact Map ...................................................................................................................................... 9 Stakeholders and Outcomes Excluded from the SROI Model................................................................12 Evaluating and Monetising Outcomes ................................................................................................13 Short-term Outcomes .......................................................................................................................15 Reduced stress and burden for family members/guardians .................................................................15 Increased family connections and relatedness ....................................................................................16 Medium-term Outcomes ...................................................................................................................17 Reduced school conduct problems ....................................................................................................17 Reduced time off work from parents/guardians ..................................................................................18 Reduced need for additional support in education and related services ................................................18 Long-term Outcomes .......................................................................................................................20 Reduced burden on government social services .................................................................................20 Increased earnings from reduced absences at school and higher educational achievement....................22 Improved wellbeing resulting in improved physical and mental health outcomes ..................................23 Reduced burden on the justice department and police services ...........................................................25 Reduced costs from school exclusions (re-engaging participants) ........................................................26 Impact ...............................................................................................................................................27 Attribution .......................................................................................................................................27 Displacement...................................................................................................................................27 Deadweight .....................................................................................................................................28 Duration and Drop-off ......................................................................................................................28 Calculating SROI .................................................................................................................................31 Results ..................................................................................................................................................32 Base case results ................................................................................................................................32 Scenarios............................................................................................................................................33 Sensitivity analysis ..............................................................................................................................35 Future Recommendations .......................................................................................................................37

Copyright Š Costello Medical Consulting Ltd

2

Noise Solution – Social Return on Investment Report July 2019

References.............................................................................................................................................42 Appendix 1: Participant Interviews ...........................................................................................................44 Noise Solution Participant ....................................................................................................................44 Parent of Noise Solution Participant ......................................................................................................45 Tutor of Noise Solution Participant .......................................................................................................47 Appendix 2: Excluded Outcomes from SROI Model ....................................................................................49 Appendix 3: Academic Level Descriptions .................................................................................................51

Copyright © Costello Medical Consulting Ltd

3

Noise Solution – Social Return on Investment Report July 2019

Introduction Noise Solution Noise Solution is a social enterprise based in the East of England, delivering personalised, one-to-one music mentoring programmes to young people facing challenging circumstances. Participants receive 10 weekly two-hour sessions with a music professional, taking place anywhere from participants’ houses to a local recording studio, in which they are given the support and freedom to co-create music. A digital community complements the sessions, allowing participants to document, reflect on and share their journey with family and friends via a bespoke social media service. The programme is underpinned by Self Determination Theory, and aims to grow participants’ feelings of autonomy, competency and relatedness.1 Supported by the relationship with their professional musician mentor, the focus is placed on the participants’ development of musical skills to create quick, tangible feelings of success in a skill deemed societally complicated. The result is improved participant wellbeing as seen through statistically significant changes in a National Health Service (NHS) validated wellbeing scale, as well as increasing confidence, self-belief, motivation, and positive social engagement, found not only in the participants but often in their mentor and family members.

The Importance of Wellbeing Improved wellbeing has been empirically shown to improve education, engagement, social and health outcomes, ultimately impacting individual quality of life and reducing the financial burden on governmental services. As outlined in Glenister (2017), these extend not only to services dedicated to mental health, but education and physical health also.2 For example, the link between improved wellbeing and physical health can be seen in Boehm and Kubzansky’s 2012 study, which highlights the association between positive mental wellbeing and cardiovascular health.2, 3 Indeed, multiple sources have shown good levels of wellbeing to have a directly positive impact on both physical and mental health.2, 4, 5 The Department for Education also supports the notion of improved wellbeing leading to improvement in education, highlighting evidence from Gutman and Vorhaus (2012) who state that “children with higher levels of emotional, behavioural, social, and school wellbeing, on average, have higher levels of academic achievement and are more engaged in school, both concurrently and in later years.”2, 6

Social Return on Investment (SROI) An SROI is an outcomes-based framework that enables organisations to understand, quantify and monetise the social, environmental and economic value created through their services. The final output is in the form of an SROI ratio, indicating the amount of social, environmental and economic value (expressed in cost units, £) delivered per £1 investment. Where available, financial proxies derived from published literature are often used to monetise outcomes. These have been fully detailed in this report.

There are seven main principles behind the SROI method, as set out by the SROI Network.7 These are described in Table 1.

Table 1: Seven principles of the SROI method7

Principle

Involvement of stakeholders

Understand what changes

Value the things that matter

Copyright © Costello Medical Consulting Ltd

Description A clear establishment of who the stakeholders are, as well as their consultation throughout the analysis to ensure the value of the programme evaluated, and the way is it measured, is informed by those directly affected.

An evaluation of how both positive and negative change is created by the programme, including those which are unintentional.

The use of financial proxies to give a value to outcomes generated by the programme, even for those which are not traded in markets and thus do not have a direct financial worth.

4

Noise Solution – Social Return on Investment Report July 2019

Only include what is material

Determining which inputs and outcomes should be included in the model to give a true and fair representation of the programme’s value.

Do not over-claim

A correct attribution of outcomes, recognising other factors which may contribute to those outside of the programme’s services.

Be transparent

An honest account of when the SROI analysis may be considered accurate and showing that it will be reported to and discussed with stakeholders.

Verify results

Seeking independent validation by an appropriate source to remove subjectivity.

The SROI principle framework has been employed throughout in this report.

Objectives for the SROI The SROI ratio produced, indicates the value generated by Noise Solution per £1 investment. Whilst the focus of this ratio is placed on social value, as opposed to purely financial, it is not intended as a comprehensive outlook of Noise Solution’s impact. The use of monetary units usefully conveys social value in a widely accepted, tangible format, but it is important to note that not all outcomes have been incorporated, including those which could not be monetised. The SROI therefore complements an evaluative assessment of the social value generated by, and importance of, Noise Solution’s work.

Costello Medical is assisting Noise Solution in identifying the value of their services, through use of an SROI analysis. The aim of this report, and accompanying model, is to communicate this value to both internal and external stakeholders. It has been populated with data from the literature, with the intention of being replaced by data directly from the Noise Solution programme in the future.

This report details the process used to create the SROI model, including the identification of relevant outcomes and stakeholders, SROI methodology and sources used. It also details the model results and presents a summary of future recommendations that may be used to improve the analysis in the future. It is hoped that this evidence can be used as a valuable addition, along with qualitative evaluations, to support future referrals to and funding of the programme.

Copyright © Costello Medical Consulting Ltd

5

Noise Solution – Social Return on Investment Report July 2019

Methodology As mentioned in the ‘Social Return on Investment (SROI)’ section above, there are seven main principles behind the SROI framework. Where appropriate, each principle has been used to guide the development of the SROI model, with specific examples provided below: •

Involvement of stakeholders Establishing the outcomes generated by Noise Solution was primarily informed by discussions with key stakeholders – namely the participants, immediate family/guardians, and mentors – that took place with Costello Medical or Noise Solution directly.

•

Understand what changes Key outcomes captured by the model involved looking at relevant changes in participants over the duration of their programme.

•

Value the things that matter Informed by published financial proxies, the SROI model considers not only the benefits and costs to stakeholders, but also to the wider population through governmental-related costs.

•

Only include what is material Attribution of outcomes to other services, and consideration of which outcomes would be present in the absence of Noise Solution (‘deadweight’), have been explored.

•

Do not over-claim

In addition to attribution and deadweight, displacement and drop-off (see the’ section for further details) have been employed to avoid overclaiming impact. Where estimates have been made, these have been conservative and clearly stated throughout.

•

Be transparent

All sources used to inform both outcomes and financial proxies have been detailed in this report. Where publications were used, these have been detailed in the There were also additional outcomes for which it was not possible to assign monetary values for, either due to lack of data, or irrelevance of available financial proxies. These have been excluded from the model and are detailed in Appendix 2: Excluded Outcomes from SROI Model.

Evaluating and Monetising Outcomes section. All decisions (and the accompanying rationale) likely to affect the outcome of the SROI have also been detailed. A one-way sensitivity analysis was also conducted, to identify the inputs which the SROI was most sensitive to.

•

Verify results We recognise the subjectivity of SROI framework, and as a result have used multiple sources to justify the inclusion and exclusion of all outcomes and costs. Where possible, stakeholders have been used to verify outcomes and their association with Noise Solution’s services.

The methodology employed to construct the SROI model is detailed in each of the succeeding headings.

Specifically, this section is split into the following sections: •

Scope and stakeholders

•

Investment

•

Outcomes and evidence

•

Impact

•

Calculating SROI

Copyright © Costello Medical Consulting Ltd

6

Noise Solution – Social Return on Investment Report July 2019

Scope and Stakeholders Time Frame of SROI The SROI focusses on participant outcomes that are generated within four years of completing the programme. Although it is recognised that some outcomes may be sustained longer than this duration, given the lack of data and in order to not over-claim the benefits, a shorter time horizon has been employed. Further details on the duration that outcomes are assumed to last in the model are provided in the Reduced costs from school exclusions (re-engaging participants) Outcomes relating to young people re-engaging with education, training, employment or voluntary placements are included across the short-, medium and long-term in the Impact Map. As participants are expected to gain confidence, competency and achievement in the short-term, and hence re-engagement in the medium-term, there is an expected reduction in the proportion of participants who have previously been excluded from schooling, to re-engage (either in mainstream schooling, or other means). This outcome has been captured by capturing related costs that have not already been included in the model, from a report conducted on the cost of school exclusions. To avoid double-counting, the costs associated with the previous long-term outcomes are not included (e.g. job seeker’s allowance, increased earnings, reduction in crime and physical/mental wellbeing).

Although data for this indicator are not routinely being collected, Noise Solution has provided estimates on the proportion of patients who have been excluded from school when beginning Noise Solution, and similarly, the proportion of these patients who re-engage with some form of education, training, employment or voluntary placement following Noise Solution.

To estimate associated costs, that are not already captured in the model’s estimates, a report published by the New Philanthropy Council,28 that estimates the total cost of school exclusions in the UK, was reviewed.

Specifically, costs to the education system and social services was extracted and uprated from 2005 prices to the latest cost year. Other costs included in the report were: impact on earnings, cost to the NHS and cost of higher crime. To calculate the net value of this outcome, the sum of these costs was applied to the proportion of participants who had been excluded from school, pre Noise Solution minus those who re-engaged, post Noise Solution.

Table 26. School exclusions

Input

Value

Source/Justification

Proportion of participants who have been excluded, pre Noise Solution

22.5%

Estimate from Noise Solution – 45% excluded/suspended, with a 50/50 split

Proportion of participants who remain excluded, post Noise Solution

14%

Estimate from Noise Solution – 40% re-engage through one avenue

Table 27. Financial proxy for school exclusions

Financial Proxy Cost of managing the process of an exclusion (one-off cost)

Value

Source/Justification

£1213.26

Misspent Youth Report: Cost of Truancy and Exclusion.28

Net annual cost of alternative educational provisions

£10,484.26

Annual cost of increased use of social services

£1,706.74

Impact section.

Copyright © Costello Medical Consulting Ltd

7

Noise Solution – Social Return on Investment Report July 2019

Stakeholders Considered Several stakeholder groups were considered in the SROI model. Prior to conducting the analyses, discussions with Simon Glenister, Director of Noise Solution, and Damian Ribbans, Operation Director, were undertaken to discuss the scope of the analyses. The stakeholder groups to include in the evaluation were narrowed down, based on both relevance and availability to interview, resulting in five key stakeholder groups: •

Programme participants

•

Immediate family members/guardians (e.g. parents and siblings)

•

Musicians tutoring Noise Solution participants

•

Organisations who own music studios used by Noise Solution

•

Other related services: private and government-funded health/social/educational services (including employees), as well as the Justice Department and police services

Where possible, interviews with one person from each of these key stakeholder groups were conducted. A representative for each stakeholder group participated in an interview, with the exception for the ‘other related services’ stakeholder group, which was assessed through other stakeholders, and previously collected qualitative data collected by Noise Solution. Interviews were conducted by Costello Medical via phone, except for the programme participant who was interviewed by their Noise Solution tutor. The interviewed stakeholders were chosen due to their willingness to participate and mutual connection to the same Noise Solution participant, a graduate of the programme. Interviewees were ensured their answers would not be shared in a way that is recognisable with Noise Solution or its staff, to encourage honest and unbiased responses.

The interviews aimed to identify and confirm key outcomes of interest for the SROI model, as well as to identify outcomes to consider for future data collection and any relevant stakeholders who were not currently considered in the model. Suggestions for future improvement to the programme were also encouraged, to identify outcomes not already being met by Noise Solution. Interview questions were pre-defined, however special care was taken to keep questions broad to allow a discussion format and avoid biasing answers.

Due to the importance of considering all five groups, including ‘other related services’, anonymised interviews with three separate health care professionals (working with Noise Solution participants in a capacity independent to Noise Solution itself), previously conducted by Noise Solution prior to this project, were retrospectively assessed. These interviews were originally conducted to understand the impact of Noise Solution’s services (and in particular the blogging platform) on the participants, and how this fed into the health care professional’s perception of Noise Solution’s impact on both themselves and the participant, and so were deemed suitable for identifying and confirming stakeholder outcomes of interest for the SROI model.

However, outcomes pertaining to the ‘other relative services’ group were eventually not included in the SROI (see Table 2 for further details).

The interview questionnaires can be found in Appendix 1: Participant Interviews. A brief description of the aims and objectives of each are as follows: •

Noise Solution participant Predefined questions were asked to investigate the main changes the participant perceived as occurring in their life, resultant of participating in Noise Solution. This included the change in their motivation levels and engagement with others, as well as their outlook for the future. To identify whether the changes can be attributed to Noise Solution, the participant was also asked to reflect on what their life might look like now, should they not have had the opportunity to participate.

•

Parent/Guardian of Noise Solution participant Predefined questions were asked which aimed to explore the perceived impact of Noise Solution on the participant, as well as to immediate family members, friends and the broader community. The relevance of the interviewee to the SROI model was also assessed, through investigation of participant’s history, including other services used and degree of involvement in the programme. This ensured the conclusions of the interview remained generalisable and relevant to the wider

Copyright © Costello Medical Consulting Ltd

8

Noise Solution – Social Return on Investment Report July 2019

stakeholder population. Perceived impact on the participant was investigated through questions relating to social, academic and emotional changes seen. Impact to immediate family members was evaluated through questions relating to changes in their social and professional lives. Comparisons of Noise Solution to other used services were also investigated, to help determine the attribution of outcomes to other services.

•

Tutor of Noise Solution participant Predefined questions were asked to assess the stakeholder relevance (as judged by degree of involvement in the programme and with the participant) and the perceived impact of Noise Solution on the participants, tutors and broader community. Perceived impact on the participant was investigated through identification of key behavioural and emotional changes. The impact on the tutors themselves was evaluated through questions relating to changes seen on both a professional and personal level, such as through job security and wellbeing. Other areas of employment were also enquired about, to help accurately assess attribution of tutor outcomes to being employed by Noise Solution as opposed to other organisations.

•

Organisations who own music studios used by Noise Solution Outcomes relating to this stakeholder were primarily assessed through discussions with Noise Solution directly, and the musicians who utilised the studios. Noise Solution were asked about the number of studios they contracted with, the terms on which the studio use was agreed, and the relationship they had established with these organisations. No discussions were undertaken with the studio organisations directly, as this was identified out of scope for the project.

•

Other related services Attempts were made to interview a representative from this stakeholder group (an employee from the National Health Service), however, these were unsuccessful. The aim of the intended interview was to assess the perceived impact of Noise Solution on other services (health/social/educational/justice).

This includes the extent to which Noise Solution complements other services, and if the burden on other services is reduced because of this. Where possible, these outcomes were instead assessed through the interview with the parent/guardian stakeholder, and discussions with Noise Solution.

Further, previously conducted interviews by Noise Solution with three health care professionals were reviewed and used to inform the Impact Map.

Alongside the previously conducted interviews with health care professionals, Noise Solution also interviewed the corresponding participant and a family member. The anonymised scripts from these, along with informal testimonials from family members of other participants, were also used to validate outcomes captured in the SROI model.

Investment The investment per participant for Noise Solution has been calculated as the cost of participating in the 10week programme (£1,827.50). This was deemed an appropriate valuation of the investment, as Noise Solution is costed on a cost-recovery basis, and therefore covers all necessary investment costs incurred.

Outcomes and Evidence Impact Map Through discussions with Noise Solution and interviews with relevant stakeholders, an Impact Map (Figure 1) was created. The Impact Map details the relationship between resources (inputs) allowing Noise Solution to deliver their services (activities), and the result of these (outputs). The outcomes of these to relevant stakeholder groups are then detailed, into three separate categories, based on the time point at which they are expected to be achieved: •

Short-term: Achieved within 3 to 6 months of completing the programme, and when maintained, may result in a medium-term or sustained outcome

Copyright © Costello Medical Consulting Ltd

9

Noise Solution – Social Return on Investment Report July 2019

•

Medium-term: Achieved within 6 to 12 months of completing the programme, and when maintained, may result in a sustained outcome

•

Sustained (long-term) outcomes: Achieved in 12+ months, but expected to be sustained for a longer period of time

This relationship is called a ‘theory of change’ and aims to break down the impact Noise Solution has on the world.

The map was developed using multiple stages and sources. The first stage involved discussions with Noise Solution, which aimed to identify all inputs (e.g. employees, funding etc.), activities and outputs (performed and generated by Noise Solution, both on an administrative and participant level). These discussions also highlighted Noise Solution’s beliefs regarding the impact of their services and were therefore additionally used to form an initial list of potential outcomes for consideration. Published literature and stakeholder discussions (see Stakeholders Considered) were then used to verify the information given by Noise Solution, and to further identify any outcomes not already captured in the map. Throughout the development process, Noise Solution were regularly consulted to ensure the information captured in the map remained valid.

Copyright © Costello Medical Consulting Ltd

10

Figure 1: Impact map

Stakeholders and Outcomes Excluded from the SROI Model After conducting interviews and creating the Impact Map, it was decided that not all stakeholders initially identified, and included in the Impact Map, were relevant for inclusion in the SROI model. These include the musicians who tutored Noise Solution participants, organisations who own musician studios used

by Noise Solution, and other related services. The reasons for excluding these groups are detailed in Table 2 below.

Table 2: Stakeholder groups excluded from the SROI model and reason for exclusion

Excluded stakeholder group from the SROI model

Tutors of Noise Solution participants

Organisations who own music studios used by Noise Solution

Other related services

Example of excluded outcome from the SROI model

Reason for exclusion from the SROI model

Support of local community through musicians’ employment

Although it was recognised that the overall impact that Noise Solution has on this stakeholder group is not negligible, after discussions with stakeholder members (see above) it was decided that outcomes relating to this are out of scope for the SROI model. This is due to a lack of outcome materiality, with some outcomes relating to this stakeholder group being difficult to quantify/monetise and other outcomes difficult to accurately attribute to Noise Solution. It was therefore decided that their inclusion may over claim Noise Solution’s benefits.

Support of local community through studio hire

To avoid over-claiming the impact of Noise Solution, organisations owning music studios were excluded from the model. This is due to the difficulty in finding an objective measure to determine studio use without Noise Solution’s custom, and so the possibility that outcomes relating to this stakeholder may have occurred without Noise Solution’s existence cannot be excluded.

Improved effectiveness of other health/social services resulting from Noise Solution acting as a catalyst for change

To avoid double counting the impact of Noise Solution on other related services (i.e. benefit to participant and service more widely), outcomes relating to the services themselves were not included in the SROI model.

Outcomes relating to employees of other related services (e.g. improved employee job satisfaction) were also excluded, as these were deemed too indirect to give firm attribution to Noise Solution. In some instances, the

Noise Solution – Social Return on Investment Report March 2019

costs to these services are used as a way of estimating the value of some outcomes achieved by the participant or family member/carer, however should not be interpreted as outcomes realised by the service.

There were also additional outcomes for which it was not possible to assign monetary values for, either due to lack of data, or irrelevance of available financial proxies. These have been excluded from the model and are detailed in Appendix 2: Excluded Outcomes from SROI Model.

Evaluating and Monetising Outcomes This section of the report details the outcomes that were included in the SROI model (based on those shown in the Impact Map – see Figure 1). The section has been separated into short-, medium- and long-term outcomes, to align with the Impact Map. If short-term outcomes feed into medium-term outcomes, or medium-term outcomes feed into long-term outcomes, they are not individually valued but are discussed in the medium- or long-term outcome section respectively only (to avoid double-counting). Discounting is applied to any costs or benefits that are incurred in the future, to account for the notion that these are valued less than those incurred immediately.

For each outcome, the parameters and the respective data sources are described. To highlight inputs that have been identified as potentially replaceable with data collected by Noise Solution in the future, orange shading in the tables is used (see Table 4 for an example). Financial proxies applied to each outcome are also described. In some cases, the specific cost relating to the outcome in question is applied to calculate the value of that outcome (e.g. cost of additional healthcare resource use). In other cases, a more abstract approximation of cost is required, when no definitive value is available (e.g. the cost of stress, or family connections). Further details on recommendations for collecting this data are provided in the

Copyright Š Costello Medical Consulting Ltd

13

Future Recommendations section.

Whilst a variety of literature sources have been employed in the model, there are three key data sources that are used throughout. A detailed description of each of these data sources is provided in Table 3.

Table 3. Key data sources to evaluate outcomes

Source

Description These reports detail the results from surveys regarding the prevalence of behavioural disorders amongst 5 to 15-year olds, and the associations of these with demographic, socio-demographic, socio-economic and social functioning characteristics.

The surveys were carried out at multiple timepoints. The 1999 report results of the original survey, whilst the 2004 report updates these, highlighting any changes observed since the 1999 survey was conducted.

The 2017 report builds on both previous reports, and also provides information for up to 19 years of age.

The Mental Health of Children and Adolescents in Great Britain (1999, 2004) 8, 9 and The Mental Health of Children and Young People (2017)10

Youth Cohort Study & Longitudinal Study of Young People in England (2007 and 2009), Department for Children, Schools and Families11, 12

The most recent data were prioritised for use in the model, meaning where possible, 2017 data were utilised. However, data from the 1999 report were sometimes used due to more granular reporting of data for particular outcomes. No data were used from the 2004 report, as all were updated in the 2017 dataset.

An overarching assumption is made throughout the model based on the results from these surveys. Based on discussions with Noise Solution, it was deemed appropriate to assume that participants could be accurately represented by assuming an equal split between having either an emotional or conduct disorder. A limitation of the surveys is the difference in population, with the participants of Noise Solution being over 10 years old, however in the absence of other data this was deemed a suitable proxy. Outcomes for participants showing a significant improvement in wellbeing (see Row 5) following Noise Solution are modelled as equivalent to those with no behavioural disorder (as reported in these surveys).

These statistical bulletins detail the responses of young people to two longitudinal surveys: The Youth Cohort Study (YCS) and the Longitudinal Study of Young People in England (LSYPE). The surveys were answered by a cohort of children who were in Year 9 during the 2003/2004 academic year. LSYPE respondents were first interviewed in 2004, and YCS respondents in 2007. The bulletins detail the behaviours, experiences, attitudes and outcomes of young people and their families, such as remaining in education at aged 16/17 based on educational achievement and intention two years prior, as well as demographic and socio-economic variables.

The most recent data, given by the 2009 bulletin, were prioritised for use as inputs for the model. Where this was not possible, data from the 2007 bulletin were used.

Post-16 Education: Highest Level of Achievement by Age 25 (2018), Department for Education13

This report describes the different routes of post-16 education taken by young people in England as identified by the Longitudinal Education Outcomes (LEO) Study. The report uses a cohort of young people taking GCSEs in 2004/05, measuring their educational achievement at ages 18, 23 and 25. Labour market outcomes (e.g. employment and use of benefits) were also observed in the 2016/17 tax year.

Noise Solution – Social Return on Investment Report April 2019

Data provided by Noise Solution

Questionnaires are given to Noise Solution participants, by Noise Solution themselves, before and after intervention with the programme. The questionnaires aim to assess whether participants underwent a meaningful positive change in wellbeing during the programme, as defined by a difference of >2 in the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS) score.14, 15 The questionnaires additionally record demographics such as gender and age.

Data on 84 participants has been recorded and was used in the model to inform age demographic inputs (proportion of participants <18 years) and number of participants showing a meaningful change in wellbeing after engaging with Noise Solution.

ONS, Office for National Statistics; LEO, Longitudinal Education Outcomes; LSYPE, Longitudinal Study of Young People in England; YCS, Youth Cohort Study.

Short-term Outcomes Reduced stress and burden for family members/guardians A short-term outcome identified in the Impact Map is the reduction in stress and reduced burden experienced by family members, resultant of the participant’s behavioural problems. This outcome estimates the proportion of parents/guardians and siblings experiencing increased stress/caregiver burden, pre and post Noise Solution, and applying the cost of a counselling intervention, to represent the value with and without Noise Solution.

Separate calculations were made for parents and siblings, each of which are described in turn. The proportion of siblings affected by the participant’s behaviour pre Noise Solution was based on reported data from the Mental Health of Children and Adolescents 1999 study.8 Specifically, the answers to ‘child’s problems cause difficulties with other family members’ for children with emotional and conduct disorders was applied as a proxy for the proportion of Noise Solution participants whose siblings were affected by their behaviour. Post Noise Solution values were calculated assuming a proportional reduction based on Noise Solution participants who showed a meaningful improvement in wellbeing.

For parents/guardians, it was assumed that prior to Noise Solution, all parents/guardians showed increased stress resultant of child’s behaviour, in the absence of relevant data. Stress and caregiver burden post Noise Solution was calculated in the same way as for siblings, with a proportional reduction based on Noise Solution participants showing a meaningful improvement in wellbeing. The average number of parents/guardians and siblings per participant (necessary for applying the cost per participant) were calculated based on data published from the ONS. Error! Reference source not found.Table 4 summarises the described inputs used to calculate the proportion of siblings who are impacted by the participant’s behaviour, pre and post Noise Solution.

The financial proxy associated with this outcome was the average cost per counselling intervention, as reported in the PSSRU 2018 (see Table 5).16 Specifically, the cost applied was for interventions ranging from 8 to 12 counselling sessions (i.e. short- to medium-term), at a value of £1092 per counselling intervention (i.e. per family member affected). It should be noted that this cost is applied as a representative approximation of the ‘value’ of reduced stress and caregiver burden, rather than the actual cost that would be expected to be incurred by family members/guardians experiencing increased stress. Therefore, should a more appropriate approximation be identified (e.g. cost of a stress-management course) this could be altered in the model.

Table 4. Estimating the reduced stress and burden for family members/guardians

Input

Value

Source/Justification

Proportion of siblings affected by child’s behaviour – for children with emotional disorders

19%

Mental Health of Children and Adolescents, 19998

Copyright © Costello Medical Consulting Ltd

15

Noise Solution – Social Return on Investment Report April 2019

Proportion of siblings affected by child’s behaviour – for children with conduct disorders

35%

27%

The model assumes that Noise Solution participants either have an emotional (50%) or conduct (50%) disorder. Therefore, the proportion of siblings affected is the average of the two inputs above.

Proportion of siblings affected, post Noise Solution

12%

The model assumes that all siblings of participants who do not show a significant improvement in wellbeing remain affected, if already in an affected state.

Average number of siblings per participant

1.36

Families with dependent children, by number of children, 1999-2017, ONS, 201817

Proportion of parents/guardians affected, pre Noise Solution

100%

Assumption

Proportion of parents/guardians affected, post Noise Solution

45%

The model assumes that all parents of participants who do not show a significant improvement in wellbeing maintain affected, if already in an affected state.

Average number of parents/guardians per participant

1.78

Families and Households, ONS, 201718

Proportion of siblings affected, pre Noise Solution

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 5. Financial proxy for reduced stress and burden

Financial Proxy Cost per counselling intervention

Value

Source/Justification

£1,092.00

PSSRU 201816

PSSRU, Personal and Social Services Research Unit.

Increased family connections and relatedness Another short-term outcome identified in the Impact Map is the increased family connections and relatedness to participants that is developed through understanding and acknowledgement. It has been populated with placeholders, in the absence of suitable data identified in the literature or collected by Noise Solution.

Increased family connections have been valued with the (avoided) cost of a parenting intervention that had a similar aim.

It was assumed that after participating in Noise Solution, the proportion of participants who show a significant improvement in wellbeing, after participating in Noise Solution (see Table 6) was equivalent to those who would feel increased relatedness to the participant.

The financial value applied to this indicator was estimated by applying the average cost of parenting interventions for the prevention of conduct disorders designed to improve parent-child relationships (see See Future Recommendations for further details.

Table 7).16 Parents who reported increased relatedness to their child were assigned this cost. The net cost calculated should not be interpreted as costs literally incurred by these parents, as with the previous outcome ‘Reduced stress and burden for family members/guardians’, but rather the value of the outcome. This financial proxy was deemed appropriate, given the intended outcome of such a programme is similar to the outcome achieved by Noise Solution.

Table 6. Parents/guardians reporting increased relatedness to participant

Input

Value

Source/Justification

Proportion of parents/guardians who reported increased relatedness to participant

55%

The model assumes that this is equal to the proportion of participants who show a significant improvement in wellbeing after Noise Solution.

Copyright © Costello Medical Consulting Ltd

16

Noise Solution – Social Return on Investment Report April 2019

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 7. Financial proxy for increased family connections and relatedness

Financial Proxy Cost of a parenting intervention for the prevention of conduct disorders, designed to improve parent-child relationships

Value

Source/Justification

£1349.00

PSSRU 201816

PSSRU, Personal and Social Service Research Unit.

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Medium-term Outcomes Reduced school conduct problems As participants develop sense of competency in a new area, and increased engagement in pro-social activities and behaviours in the short-term, there is expected to be a reduction in school conduct problems, such as detentions in the medium-term. These short and medium-term outcomes from the Impact Map have been calculated by estimating the proportion of Noise Solution participants that would receive behavioural complaints that disrupt classroom teaching, both pre- and post-participation in Noise Solution, and monetising this to generate a total value of a situation with Noise Solution and without Noise Solution. As this outcome is specific to school behaviour, it is applied to school-aged participants only.

In the absence of any data identified in the literature or collected by Noise Solution, the reduction in school conduct problems has been based on assumptions. It has been assumed that before Noise Solution, 100% of participants receive behavioural complaints at school. Post Noise Solution, it is assumed that participants who show a significant improvement in wellbeing no longer receive any behavioural complaints, while all other participants do. These inputs are summarised in Table 8.

The financial proxy used to calculate the value of this outcome was the cost of a school-based social and emotional learning programme, as reported in the PSSRU 2018 (see Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 9).16 Table 8. Estimating reduced behaviour complaints in school

Input

Value

Source/Justification

Proportion of children with behaviour complaints in school, before Noise Solution

100%

Assumes all participants receive behavioural complaints.

54%

Assumes that the proportion of participants who show a significant improvement in wellbeing no longer receive behavioural complaints, while all other participants do.

Proportion of children with behaviour complaints in school, after Noise Solution

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 9. Financial proxy for reduced school conduct problems

Financial Proxy Cost of a school-based social and emotional learning programme

Value

Source/Justification

£151.00

PSSRU 201816

PSSRU, Personal and Social Services Research Unit.

Copyright © Costello Medical Consulting Ltd

17

Noise Solution – Social Return on Investment Report April 2019

Reduced time off work from parents/guardians As participants show increased engagement in pro-social activities and behaviour in the short-term, there is anticipated increased labour productivity from parents due to increased/return to work. The value of this outcome was approximated by calculating the average number of days absent from work pre, during and post Noise Solution, and applying a pro-rata average wage for each day absent, across both parents.

In the absence of data from Noise Solution, it was assumed that pre Noise Solution, one parent/guardian per participant was absent one day per month, resulting in an annual estimate of 12 days. During the 10-week duration of Noise Solution, it was assumed that one parent/guardian per participant misses a quarter of a day each session, resulting in an estimate of 6 days (quarter of a day, twice a week for 10 weeks). After Noise Solution, it was assumed that the parents of participants who show a significant improvement in wellbeing no longer have any absent days from work related to caring for the participant, while parents of other participants remain at pre Noise Solution levels. These inputs are summarised in Table 10.

This outcome has been monetised by valuing the loss of productivity resulting from parents/guardians being absent from work. The financial proxy for this outcome is the national average annual wage in the UK (see Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 11). Without Noise Solution, the annual loss of productivity is based on the annual number of days missed out of a year. With Noise Solution, the annual loss of productivity, in the first year, is calculated assuming that Noise Solution takes place over the first 12 weeks of the year, and the remaining 40 weeks of the year is calculated assuming the post Noise Solution rate of absenteeism. In subsequent years, the annual loss of productivity is based on the post Noise Solution rate of absenteeism only.

Table 10. Loss of productivity for parents/guardian

Input

Value

Source/Justification

Annual numbers of days absent from work, caring for participant pre Noise Solution

12.00

Assumes one day per month

Average numbers of days absent from work, caring for participant during Noise Solution

6.00

Assumes a quarter of a day is missed each session, based on one session per week for 10 weeks

2.68

Assumes that the parents of participants who show a significant improvement in wellbeing no longer have any absent days from work, while other parents of participants remain at pre Noise Solution levels.

Annual numbers of days absent from work, caring for participant post Noise Solution

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 11. Financial proxy for loss of productivity

Financial Proxy National average annual wage

Value

Source/Justification

£29,869.61

ONS 201919

ONS, Office for National Statistics.

Reduced need for additional support in education and related services A medium-term outcome identified in the Impact Map is the reduced need for additional support in education and other related services. This outcome has been valued by estimating the use (and costs) of educational support services in the 12 months pre- and post-participation in Noise Solution, allowing a calculation of a situation with and without Noise Solution. As this outcome is specific to school, it is only calculated for schoolaged participants.

Three educational services were captured in the model; educational psychologist, special needs support and classroom assistance, based on resources cited in literature.10, 20 The proportion of participants in contact with an educational psychologists, and/or with special educational needs, is based on the Mental Health of Children and Young People 2017 survey,10 stratified by type of behavioural disorder (see Table 12). In the absence of

Copyright © Costello Medical Consulting Ltd

18

Noise Solution – Social Return on Investment Report April 2019

estimates for classroom assistants, this was assumed equal to the proportion of children accessing special educational needs (SEN) support. Pre Noise Solution, participants are assumed equally distributed between emotional and conduct disorders. Post Noise Solution, the proportion of participants showing a significant improvement in wellbeing were assigned resource use estimates equivalent to the ‘No behavioural disorder’ children. Mean use of each service was derived from Romeo 2006,20 a paper estimating the cost of severe antisocial behaviour in children, which collected service use data for children who had been referred to mental health services in the UK. It was assumed that post Noise Solution, the mean use of each service remained the same (within each disorder), in the absence of alternative data.

Costs for each service were derived from the PSSRU and the Education Endowment Foundation, and are summarised in Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 13.16, 21 The cost of an educational psychologist was assumed equal to a paediatric occupational therapy visit. Special needs support and classroom assistance hourly rates were calculated based on the mean wage of a teaching assistant in the UK, assuming 195 working days and 6.5 hour days.

Table 12. Resource use of education-related services

Behavioural Disorder

Emotional

Conduct

No behavioural disorder

Pre Noise Solution

Post Noise Solution

Input

Value

Proportion of children who sought educational support services in the last 12 months

20.10%

Proportion of children with SEN

26.80%

Proportion of children who accessed classroom assistance

Assumed equal to SEN

Proportion of children who sought educational support services in the last 12 months

31.90%

Proportion of children with SEN

42.40%

Proportion of children who accessed classroom assistance

Assumed equal to SEN

Proportion of children who sought educational support services in the last 12 months

1.90%

Proportion of children with SEN

6.10%

Proportion of children who accessed classroom assistance

Assumed equal to SEN

Proportion of children who sought educational support services in the last 12 months

26.00%

Proportion of children with SEN

34.60%

Proportion of children who accessed classroom assistance

Assumed equal to SEN

Proportion of children who sought educational support services in the last 12 months

14.81%

Proportion of children with SEN

21.37%

Proportion of children who accessed classroom assistance

Assumed equal to SEN

Copyright © Costello Medical Consulting Ltd

Source/Justification

Mental Health of Children and Young People in England, 201710

Assumes participant either have an emotional (50%) or conduct (50%) disorder. Resource use of participants pre Noise Solution is the average of each of the inputs above.

All participants who show a significant improvement in wellbeing are applied resource use of the ‘No behavioural disorder’ children

19

Noise Solution – Social Return on Investment Report April 2019

All

Educational psychology annual use (hours)

2.00

Special needs support annual use (hours)

8.70

Classroom assistance annual use (hours)

108.71

Romeo 2006.20 Special needs support calculated from mean use of 3 months, 40 minutes per week. Classroom assistance calculated from mean use of 5 months, 1 hour per school day.

SEN, special education needs.

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 13. Financial proxy for reduced need for educational support services

Financial Proxy

Value

Source/Justification

Educational psychologist

£84.00

PSSRU 201816 - Paediatric occupational therapy, cost per hour

Classroom assistance and SEN

£14.20

Education Endowment Foundation16 – cost per hour based on average annual salary of teaching assistant (£18,000) with 195 working days per year, and 6.5 hours per day.

PSSRU, Personal and Social Services Research Unit; SEN, special education needs.

Long-term Outcomes Reduced burden on government social services As participants show re-engagement with education, training or employment, in the medium-term, there is an expected reduced burden on government social services in the long-term, that would otherwise be spent to support these youth. The value of this outcome has been calculated by linking school truancy in Year 11 to the duration of being not in education, employment or training (NEET) at 18 years, and therefore being eligible for the Job Seeker’s Allowance (JSA). These data have been used to generate a scenario with and without Noise Solution, and hence calculate the total value generated for this outcome.

Currently, data on truancy of Noise Solution participants are not collected. Therefore, truancy of the Noise Solution population was estimated from the literature, with inputs summarised in Table 14. For school-aged participants, it was assumed that a representative population are children with emotional and/or conduct disorders, as reported in the Mental Health of Children and Young People 2017 study.10 This survey reported truancy status for children with either of these behavioural disorders, and these data were used to estimate the proportion of participants who are truant prior to participating in Noise Solution. To estimate the proportion of participants who would be truant after participating in Noise Solution, the proportion of schoolaged participants who showed a significant improvement in wellbeing (based on data provided by Noise Solution) were assumed equivalent to children with no behavioural disorder, while all other participants remained at pre-Noise Solution levels.

The probability and duration of NEET at 18 years for participants ‘with’ and ‘without’ Noise Solution was derived from these truancy estimates, based on data from the Youth Cohort Study & Longitudinal Study of Young People in England 2009 (see Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 15 and Table 16).12 The study reported annual duration of NEET in ranges of 0 months, 1-12 months and 12+ months, dependent on truancy status. In the SROI model, a 6-month and 12-month estimate was conservatively applied for the 1-12 and 12+ months range respectively. In the absence of specific literature available on NEET duration for non-school-aged participants pre Noise Solution, these participants were assumed to be NEET for the same duration as school-aged participants, without Noise Solution. Post-Noise Solution for non-school-aged participants however were based specifically on the proportion of non-schoolaged participants (as opposed to school-aged participants) who showed a significant improvement in wellbeing.

Copyright © Costello Medical Consulting Ltd

20

Noise Solution – Social Return on Investment Report April 2019

The financial proxy associated with this outcome was the weekly JSA for 16-24 year olds, applied to each group of participants (school-aged and non-school-aged, with and without Noise Solution) for the mean duration participants are NEET annually (see Table 17). This age range was deemed appropriate given the majority of Noise Solution participants are under 24 years.

As this outcome is realised in the future for school-aged participants, discounting was applied to both the with and without Noise Solution values. The value per year was discounted at a constant rate of 3.5%, for the number of years between the mean age of a school-aged participant (14 years) and the age at which the outcome is realised (18 years). No discount was applied to non-school-aged participants, as it was assumed that the value of Noise Solution would be realised immediately.

Table 14. Base case inputs for truancy, based on behavioural disorder

Input

Value

Emotional disorder

9.70%

Conduct disorder

11.20%

No behavioural disorder

0.80%

Truancy behaviour

Participants with truant behaviour, pre-Noise Solution Participants with truant behaviour, post Noise Solution

Proxy input for participants > 18 years ‘truant behaviour’, post Noise Solution

Source/Justification Mental Health of Children and Young People in England, 201710

10.45%

Assuming participants have either an emotional (50%) or conduct (50%) disorder.

5.97%

Assuming all participants with a significant improvement in wellbeing show a reduction in truant behaviour (equal to truancy of no behavioural disorder).

3.40%

A ‘proxy’ value is calculated for participants >18 years, based on the age-specific probability of showing a significant improvement in wellbeing.

It is a proxy, as these participants are not school-aged, and therefore cannot be ‘truant’, however is used similarly for subsequent calculations.

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 15. Duration of NEET based on truancy, from literature

Annual duration of NEET (at 18 years) Input

Annual weeks spent NEET

0

1-12 months

12+ months

Truant

38.00%

29.00%

33.00%

24.78

Not truant

76.00%

20.00%

4.00%

7.31

Source/Justification Youth Cohort Study & Longitudinal Study of Young People in England 200912

NEET, not in education, employment or training.

Table 16. Weeks spent NEET annually by participant group

Input

Annual weeks spent NEET

Source/Justification

School-aged participants, without Noise Solution

9.13

School-aged participants, with Noise Solution

8.35

Calculated from truancy in Table 14 (based on behavioural disorder) and mean duration for truant/non truant children in Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future.

See Future Recommendations for further details.

Copyright © Costello Medical Consulting Ltd

21

Noise Solution – Social Return on Investment Report April 2019

Table 15 Non school-aged participants, without Noise Solution

9.13

Assumed the same as schoolaged participants without Noise Solution.

Non school-aged participants, with Noise Solution

7.90

Calculated as per school-aged participants, with proxy input for > 18 years.

NEET, not in education, employment or training.

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 17. Monetising the reduced burden on government services

Financial Proxy

Value

Source/Justification

JSA Weekly allowance for 16-24 years

£57.90

Department for Work and Pensions, 201822

JSA, Job Seeker’s Allowance.

Increased earnings from reduced absences at school and higher educational achievement A second long-term outcome resulting from re-engagement with activities, education, training or employment, is the expected increased educational attainment and opportunities. The value of this outcome has been calculated by projecting the higher educational achievement and subsequent average increased earnings participants would receive in the future with participation in Noise Solution, compared to without Noise Solution. As the outcome specifically relates to school behaviour (e.g. school absenteeism), this outcome was applied to school-aged participants only.

Reduced school absenteeism was estimated using truancy data from the Mental Health of Children and Young People, England 2017 study as per the previous outcome (described in Table 14).10 Truancy estimates were then used to estimate General Certificate of Secondary Education (GCSE) scores, based on data reported in the Youth Cohort Study & Longitudinal Study of Young People in England, 2007.11 Specifically, participants were distributed across three GCSE groupings, based on truancy: GCSE D-G/None, 1-4 GCSE A-C and 5+ GCSE A-C. Longitudinal data reported in the Post-16 Education 2018 bulletin was used to predict the academic level achieved by participants at age 23, stratified by GCSE grouping.13 These inputs are summarised in Table 18.

Increased earnings were calculated by estimating the median annual salary at age 26, based on the academic level achieved at age 23, using longitudinal data from the Post-16 Education 2018 bulletin (see Table 19).13 Due to the data reported, any income earned prior to age 26 was not included, which is likely to underestimate the net benefit. Salaries were inflated to the most recent cost year from 2016/17, using the ONS average pay growth rate of 3.4%.19 Separate estimates were calculated for truant and non-truant children, allowing a comparison between a situation ‘with Noise Solution’ and ‘without Noise Solution’.

As this outcome is realised in the future (age 26), the net additional income earned per year was discounted at a constant rate of 3.5%,23 for the number of years between the median age of school-aged participants (age 14) and the age at which the outcome is realised (age 26).

Table 18. Predicted academic achievement based on truancy and GCSE scores

≤ GCSE DG

1-4 GCSE A-C

5+ GCSE A-C

Truant

30.20%

25.60%

44.20%

Non-truant

14.00%

19.00%

67.00%

< L2

33%

4%

0%

Input GCSE level

Copyright © Costello Medical Consulting Ltd

Source/Justification Youth Cohort Study & Longitudinal Study of Young People in England, 200711

22

Noise Solution – Social Return on Investment Report April 2019

Academic level achieved at age 23

L2

40%

26%

3%

L3

21%

42%

21%

L4/5

3%

6%

5%

L6

3%

20%

61%

L7

0%

2%

10%

Post-16 Education, 201813 (columns sum to 100%)

GCSE, general certificate of secondary education.

Please note that details of academic levels are provided in Appendix 3.

Table 19. Financial proxy for increased earnings

Financial Proxy Median annual earnings at age 26 Average pay growth

Value

Source/Justification

£17,892.33 (GCSE D-G) £21,648.27 (1-4 GCSE A-C) £28,109.18 (5+ GCSE A-C)

Post-16 Education 201813

3.4%

ONS 201913, 19

GCSE, general certificate of secondary education; ONS, Office for National Statistics.

Improved wellbeing resulting in improved physical and mental health outcomes Improved wellbeing was identified as a medium-term outcome on the Impact Map, which consequently would result in improved physical and mental health in the long-term. The value of this outcome has been calculated by estimating the healthcare resource use (and associated costs) of participants in the 12 months pre- and post-participation in Noise Solution, to generate a total value of a situation with Noise Solution and without Noise Solution.

To estimate the behaviour-related healthcare resource use of Noise Solution participants, behaviour-related health service use in the last 12 months of young people (ages 5-15) with emotional, conduct and no behavioural disorder was obtained from the Mental Health of Children and Adolescents, 1999 bulletin.8 In the absence of age-specific data, it was assumed that this resource use was representative for all participants.

Health services reported on were general practitioner (GP), accident and emergency (A&E), inpatient stay, outpatient or day case, and medication prescriptions. Frequencies were reported in categories of 0, 1, 2, 3 or 4+ visits in the last 12 months. To avoid overclaiming, responses reported as 4+ were conservatively assumed to be four visits, as no upper values were provided. Medication prescription was dichotomous (yes/no). As prescriptions ranged considerably, a simplification was made in the model to apply the most common prescription reported within each behavioural disorder category to all participants, aligning with the methodology used in McCrone 2008.24 McCrone 2008 used the same data from the 1999 bulletin and assumed a daily 20 mg administration of Ritalin for all participants that reported receiving behaviour-related medication prescriptions. All inputs that were used to derive estimates of resource use are summarised in Table 20 and Table 21.

The value of this outcome was calculated by estimating the cost of each type of service use, using values provided in the Personal Social Services Research Unit (PSSRU) 2018, for both pre and post Noise Solution (see Table 22).16 In addition, the resource use identified in the Mental Health of Children and Adolescents bulletin, CAMHS community care was included based on feedback from Noise Solution. The proportion of participants accessing CAMHS pre and post Noise Solution, was estimated by Noise Solution. In absence of alternative data, it was assumed that the mean duration of engagement was 12 months. The value of this outcome was calculated by using the estimate of one year of CAMHS community care from NHS Benchmarking, for both pre and post Noise Solution. Inputs are summarised in Table 23.

Table 20. Behaviour-related health resource use, stratified by behavioural disorder

Annual Frequency (%)

Copyright © Costello Medical Consulting Ltd

Source/Justification

23

Noise Solution – Social Return on Investment Report April 2019

Behavioural Disorder

Emotional

Resource Use

0

1

2

3

4

GP

51%

10%

11%

6%

22%

A&E

75%

18%

4%

2%

1%

Inpatient

91%

8%

1%

0%

0%

Outpatient

71%

8%

6%

3%

12%

Medication

Conduct

15.20%

GP

55%

12%

11%

8%

14%

A&E

73%

20%

5%

2%

0%

Inpatient

91%

7%

1%

1%

0%

Outpatient

73%

8%

6%

3%

10%

Medication

No behavioural disorder

Mental Health of Children and Adolescents in Great Britain, 19998

14.80%

GP

65%

12%

10%

5%

8%

A&E

83%

13%

3%

1%

0%

Inpatient

95%

4%

1%

0%

0%

Outpatient

82%

8%

4%

2%

4%

Medication

0.50%

A&E, accident and emergency; GP, general practitioner.

Please note that responses reported as 4+ were assumed to be four visits only. Frequency for medication represents the proportion of children who receive prescribed medication specifically for behaviour concerns.

Table 21. Average annual resource use, pre and post Noise Solution

Annual value (pre Noise Solution)

Annual value (post Noise Solution)

GP

1.26

1.00

A&E

0.36

0.28

Inpatient

0.11

0.08

Outpatient

0.73

0.54

Medication

54.79

25.49

Input

Source/Justification Assumes that Noise Solution participants have an emotional (50%) or conduct (50%) disorder. Post Noise Solution, the model assumes that the proportion of participants showing a significant improvement in wellbeing have the resource use of children with no behavioural disorder.

Medication calculated assuming daily administration.

A&E, accident and emergency; GP, general practitioner.

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 22. Cost of health service resource use

Financial Proxy

Value

Source/Justification

GP

£33.30

PSSRU 201816

A&E

£120.00

Inpatient Outpatient

Copyright © Costello Medical Consulting Ltd

£2,880.00 £194.00

24

Noise Solution – Social Return on Investment Report April 2019

£10.92

Medication

BNF 2019 – cost of Ritalin (20mg) pack size 30

A&E, accident and emergency; BNF, British National Formulary; GP, general practitioner; PSSRU, personal and social services research unit.

Table 23. CAMHS Community Care Related Inputs

Input

Pre Noise Solution

Post Noise Solution

Source/Justification

12

12

Assumption - assumes each participant accesses a full year of community care

45%

35%

Estimate from Noise Solution

Mean duration (months) in community care Proportion of participants accessing CAMHS community care Annual cost

£3304.00

NHS Benchmarking, CAMHS Project 2018

CAMHS, Children and Adolescent Mental Health Services; NHS, National Health Service.

Reduced burden on the justice department and police services Conduct-related outcomes are included across the short- medium and long-term in the Impact Map. As participants are expected to engage in more prosocial activities and behaviours in the short-term, there is an expected reduction in conduct and offending behaviour in the medium-term. The longer-term implication is reduced burden on the justice department and police services. This outcome has been captured by estimating the frequencies (and costs) of cautions or convictions of participants, pre and post Noise Solution, allowing a calculation of a situation with and without Noise Solution.

The proportion of participants who have offended pre Noise Solution was based on estimates reported in the Mental Health of Children and Adolescents, 1999 survey, stratified by behavioural disorder.8 While this survey only covers the ages 11-15, it was assumed that young people 16-24 showed similar behaviour. This is in line with the principle of not overclaiming, as offending is likely to increase within this age bracket.

The average annual number of cautions/convictions per offender was derived from the Youth Justice (YJ) Annual Statistics for 11-18 year olds, and from the Office for National Statistics (ONS) for 19-24 year olds.25, 26 Children less than 11 years old, and older than 24 years, were not captured, due to a lack of data. However, given that the majority of participants fall within this age bracket, this is not expected to have a major effect on results. In the absence of data suggesting otherwise, it was assumed that the annual number of cautions/convictions per offender post Noise Solution is equivalent to that pre Noise Solution. Table 24 summarises the described inputs used to calculate offending probability and average number of cautions/convictions in the past 12 months, pre and post Noise Solution.

To estimate the cost of crime, estimates were required for the distribution of offences across different types of crime. For youth crime (i.e. ages 11-18), this was obtained from the YJ Annual Statistics 2016/17 of proven offences. For adult crime (i.e. ages 19-24), this was obtained from the ONS.25, 26 Unit costs were obtained from the Home Office 2018, referencing responder costs only (i.e. costs to the police and criminal justice system).27 A weighted average cost per crime was calculated weighted by the type of crime, unit cost per crime, and age group of participants (summarised in Table 25). To calculate the net value of this outcome, the weighted average cost per crime was applied to the proportion of participants with a caution/conviction in the past 12 months, and the mean number of cautions/convictions per offender, pre and post Noise Solution.

Table 24. Annual cautions/convictions by behavioural disorder, and age group

Input

Value

Source/Justification

Caution or conviction in the last 12 months – emotional disorder

9%

Caution or conviction in the last 12 months – conduct disorder

27%

Mental Health of Children and Adolescents, 1999.8 These values are specifically for children aged 11-15.

In the absence of alternative data, it was assumed

Copyright © Costello Medical Consulting Ltd

25

Noise Solution – Social Return on Investment Report April 2019

Caution or conviction in the last 12 months – no behavioural disorder

Pre Noise SolutionCaution/conviction in the last 12 months (all ages)

Post Noise SolutionCaution/conviction in the last 12 months (all ages) Annual number of cautions/convictions per offender (ages 11-18) Annual number of cautions/convictions per offender (ages 19-24)

2%

that 16-24 year olds showed similar offending behaviour.

18%

The model assumes that Noise Solution participant either have an emotional (50%) or conduct (50%) disorder. Therefore, this is calculated as the average of the two inputs above.

The model has the flexibility to apply distinct values for 11-18 year olds, and 19-24 year olds, however values are equal in the base case.

10.57%

All participants who show a significant improvement in wellbeing are applied resource use of the ‘No behavioural disorder’ children.

The model has the flexibility to apply distinct values for 11-18 year olds, and 19-24 year olds.

2.10

YJ Annual Statistics, 2016-17.25 Pre and Post Noise Solution assumed the same in the absence of data, however the model has the flexibility to apply distinct values.

4.01

ONS, Proven reoffending statistics 2016-17. Pre and Post Noise Solution assumed the same in the absence of data, however the model has the flexibility to apply distinct values.

ONS, Office for National Statistics; YJ, Youth Justice.

Orange shading indicates inputs that may be replaced with data collected by Noise Solution in the future. See Future Recommendations for further details.

Table 25. Financial proxy for reduced burden on justice department

Financial Proxy

Value

Source/Justification

Weighted average cost of crimes, by type of crimes - youth

£1529.65

Home Office 201827 (≤ 18 years)

Weighted average cost of crimes, by type of crimes - adult

£1567.58

Home Office 201827 (> 18 years)

Note – this cost is the cost to the criminal justice system only (i.e. justice department and police services for responding to crime).

Reduced costs from school exclusions (re-engaging participants) Outcomes relating to young people re-engaging with education, training, employment or voluntary placements are included across the short-, medium and long-term in the Impact Map. As participants are expected to gain confidence, competency and achievement in the short-term, and hence re-engagement in the medium-term, there is an expected reduction in the proportion of participants who have previously been excluded from schooling, to re-engage (either in mainstream schooling, or other means). This outcome has been captured by capturing related costs that have not already been included in the model, from a report conducted on the cost of school exclusions. To avoid double-counting, the costs associated with the previous long-term outcomes are not included (e.g. job seeker’s allowance, increased earnings, reduction in crime and physical/mental wellbeing).

Although data for this indicator are not routinely being collected, Noise Solution has provided estimates on the proportion of patients who have been excluded from school when beginning Noise Solution, and similarly, the proportion of these patients who re-engage with some form of education, training, employment or voluntary placement following Noise Solution.

To estimate associated costs, that are not already captured in the model’s estimates, a report published by the New Philanthropy Council,28 that estimates the total cost of school exclusions in the UK, was reviewed.

Copyright © Costello Medical Consulting Ltd

26

Noise Solution – Social Return on Investment Report April 2019

Specifically, costs to the education system and social services was extracted and uprated from 2005 prices to the latest cost year. Other costs included in the report were: impact on earnings, cost to the NHS and cost of higher crime. To calculate the net value of this outcome, the sum of these costs was applied to the proportion of participants who had been excluded from school, pre Noise Solution minus those who re-engaged, post Noise Solution.

Table 26. School exclusions

Input

Value

Source/Justification

Proportion of participants who have been excluded, pre Noise Solution

22.5%

Estimate from Noise Solution – 45% excluded/suspended, with a 50/50 split

Proportion of participants who remain excluded, post Noise Solution

14%

Estimate from Noise Solution – 40% re-engage through one avenue

Table 27. Financial proxy for school exclusions

Financial Proxy Cost of managing the process of an exclusion (one-off cost)

Value

Source/Justification

£1213.26

Misspent Youth Report: Cost of Truancy and Exclusion.28

Net annual cost of alternative educational provisions

£10,484.26

Annual cost of increased use of social services

£1,706.74

Impact To avoid overclaiming the impact of Noise Solution, the following concepts were applied:

Attribution Attribution involves the recognition that the outcomes achieved may not be due to Noise Solution alone.

Through use of a percentage, attribution estimates the proportion of the impact that is created by other services and accounts for this when calculating value of the intervention in question.

Based on the stakeholder interviews, it is understood that many participants engaging with Noise Solution simultaneously receive additional support (such as counselling or therapy). However, Noise Solution is understood to act as an enabler to the effectiveness of these alternative interventions, and therefore remains a key component to the overall success of the participant. Furthermore, the majority of participants who are referred to Noise Solution have already tried alternative interventions that have not delivered the satisfactory results. As a result, it is estimated that the proportion of outcomes observed attributable to other interventions is expected to be relatively low.

In the absence of data, 25% of each outcome has been attributed to another intervention, and therefore not included in the final SROI calculation. The model includes the flexibility for the user to vary this to 50% (more conservative) or 0% (less conservative) for all of the outcomes, or alternatively, the user can adjust it for each outcome individually.

Displacement This is the concept of one outcome displacing another. For example, if the presence of an outcome in Noise Solution’s stakeholders caused the loss of that outcome in another group (e.g. increased income for the stakeholder X results in reduced income for stakeholder Y).

Copyright © Costello Medical Consulting Ltd

27

Noise Solution – Social Return on Investment Report April 2019

Based on feedback from Noise Solution and the stakeholder interviews, it was deemed that none of the outcomes achieved are negative, and therefore, the concept of displacement is not relevant. As a result, displacement has not been included in the SROI calculations.

Deadweight Deadweight measures to what extent an outcome would have been observed if Noise Solution’s services did not exist. As with attribution, it is represented as a percentage, and is estimated through comparisons with groups unrelated to Noise Solution.

Deadweight is implicitly included in the SROI calculations as each outcome is calculated as a difference between the situations ‘With Noise Solution’ and ‘Without Noise Solution’. Therefore, any outcomes that are expected to have occurred in the absence of Noise Solution are taken into account.

Duration and Drop-off Several outcomes in the model are expected to last for more than one year. However, as time goes on, there is a possibility of declining effect, increased attribution to other services or lack of long-term data. Applying drop-off avoids overestimating the long-term impact of outcomes lasting over one year, with the percentage applied indicating the proportion that ‘drops off’ each year (i.e. is not included in subsequent calculations).

The model includes drop-off for outcomes that are expected to last more than one year, with estimates based on the long-term data reported in the literature where possible. For outcomes expected to last more than one year but no data identified for estimating drop-off, it has assumed to be 50% - common in SROI practice.7 The model includes the flexibility for the user to vary this to 50% for all inputs (more conservative), or 0% (less conservative). Alternatively, the user can adjust this input for each outcome individually. In addition, the model also includes the flexibility to calculate the SROI assuming each outcome lasts for one year only. If this option is selected, drop-off is no longer relevant, and therefore not applied in the model.

A summary of these inputs and how they are applied to each outcome in the model is shown in Table 28.

Copyright © Costello Medical Consulting Ltd

28

Table 28. Measuring Impact - Attribution, Duration and Drop-off Applied for each Outcome

Outcome

Indicator

Annual net value

Attribution

Duration (years)

Annual drop-off

Source for Duration and drop-off

Participants become reengaged with society through education, training or employment resulting in a reduced burden on government social services

Average number of weeks spent NEET in the past 12 months, pre and post Noise Solution

£278.91

25%

2

48%

Drop-off based on Impetus 2017 where 52% of NEET people remained NEET the following year.29 In the absence of data, this outcome has been conservatively assumed to only last two years.

Increased earnings resulting from reduced absences at school, and higher educational achievement

GCSE scores (based on level of truancy) to predict academic level achieved at age 23, and earnings at age 26

£346.65

25%

3

50%

Outcome has been assumed capped at age 29 (i.e. duration is 3 years) based on common age brackets in reported mean earnings.19

Wellbeing of participants improves, resulting in improved physical and mental wellbeing

Behaviour-related health services resource use in the 12 months pre and post Noise Solution

£476.22

25%

2

4%

Drop-off based on 6-month follow-up from the Mental Health of Children in Great Britain 2004 study, that showed 2% reduction in 6 months.9 Assuming a linear extrapolation to 4% at 12 months.

Duration of 2 years assumed, in the absence of long-term data.

Participant engagement in education becomes increases, reducing school conduct problems

Proportion of children with behaviour-related complaints at school, pre and post Noise Solution

£70.11

25%

4

50%

Duration based on the number of years between median age of school-aged participants, and age participants are expected to complete secondary school (i.e. 18 years).

Participants who are excluded from school become re-engaged in education, training or voluntary placements

Proportion of children who were previously excluded/suspended from school, and become reengaged

£1206.38 (first year)

25%

2

50%

Duration based on the number of years between median age of school-aged participants, and age 16, to align with the maximum age used in Misspent Youth, when calculating the total cost of school exclusions. In absence of alternative data, drop-off has been

£1097.19 (subsequent years)

Noise Solution – Social Return on Investment Report April 2019

aligned with a SROI publication of a UK school-based wellbeing programme, which assumes 50% each year.

Increased labour productivity as parents take reduced time off work

Average number of days absent from work, pre, post and post Noise Solution

£365.32

25%

2

50%

Outcome expected to be correlated to ‘Increased family connections and relatedness’ (see row below) and therefore, assumed to last for two years.

25%

2

58%

YJ statistics 2016-17 that 42.3% of youth offenders re-offend within 12 months.25 In the absence of data beyond this, outcome is assumed to only last two years.

(first year) £719.36 (subsequent years)

Reduced burden on the justice department and police services

Average number of cautions and convictions in the last 12 months pre and post Noise Solution

Reduced need for additional support in education and other services

Average educational resource use (educational psychologist, special needs support in classroom and classroom learning assistance) pre and post Noise Solution

£239.41

25%

4

50%

Duration based on the number of years between median age of school-aged participants, and age participants are expected to complete secondary school (i.e. 18 years).

Reduced stress and burden for family members/guardian

Proportion of parents/siblings affected emotionally by participants’ behaviour, pre and post Noise Solution

£1296.91

25%

2

50%

Outcome expected to be correlated to Increased family connections and relatedness (see row below) and therefore, assumed to last for two years.

Increased family connections and relatedness

Proportion of parents who report increased connectedness to child

£746.15

25%

2

50%

Duration and drop-off based on a SROI conducted in substance misuse, which used the same outcome and indicator.30

Copyright © Costello Medical Consulting Ltd

£273.05

30

Calculating SROI The SROI is calculated as a ratio of the net benefits generated through the programme, to the net investment required, and can be interpreted as a per participant value. The simple equation applied to calculate the SROI is shown below. !ROI =

Total benefit (net) Total investment (net)

The total net investment is calculated as the per-participant cost of participating the programme, as described in the Investment section of the report.

The total net benefit is the sum of each outcome that was discussed in the Outcomes and Evidence section of the report, once the net impact of each outcome had been assessed, as described in the section and summarised in Table 28. As per national evaluation guidelines, an annual discount rate of 3.5% was applied for each outcome with a duration of more than one year, to discount outcomes that occurred in the future.23

Results Base case results The net investment per participant was estimated at £1,827.50, aligning with the per-participant cost of participating in Noise Solution, for 10 weeks. The net benefit per participant was estimated at £6,109.63. As a result, the calculated SROI in the base case was 3.34 . This suggests that for every £1 invested in the programme, a value of £3.34 is generated.

The valuation of each outcome with and without Noise Solution is shown in Figure 2, represented as an annual cost per participant. It should be noted that this represents the value of each outcome prior to its impact being assessed, meaning the principles such as attribution, displacement, deadweight, duration and drop-off have not been applied.

The total net value of each outcome, including the impact assessment (i.e. applying the attribution, displacement, discounting, duration and drop-off for each outcome), is shown in Figure 2. Family stress and burden, followed by family connections and relatedness, are reported as the two outcomes generating the most value in the SROI. Improvements in physical and mental health, criminal activity, time off work from parents and education support services all generate similar value. However, these results should be interpreted as cautions, particular the former outcomes, as they are heavily dependent on assumptions and proxy data. Similarly, the true value of other outcomes may be understated (e.g. reliance on government social services) due to the conservative assumptions underpinning the with and without Noise Solution scenarios.

Figure 2. Annual cost per participant, with and without Noise Solution

Note, the annual cost per participant for ‘Reduced time off work (parents)’ and ‘Reduced school expulsions’ indicate the values in the first year. In subsequent years, the participant is assumed to have completed Noise Solution, and therefore, alternative values are calculated for the ‘With Noise Solution’ scenario.

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

Figure 3. Net benefit generated per outcome, with impact assessment

Figure 4. Scenario Results

Scenarios SROI estimates will vary depending on the assumptions employed when applying the calculations for duration, drop-off and attribution. As a result, an SROI can be thought of as a point along a sliding scale, with estimates from least to most conservative. Therefore, a number of scenarios were conducted to identify varying possible SROI estimates, varying these values in the model. The scenarios conducted, with their respective SROI estimates, are depicted in Figure 5 and summarised in Table 29. Notably, none of the

Copyright Š Costello Medical Consulting Ltd

33

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

scenarios result in an SROI less than 1, meaning the net benefit exceeds the investment in each scenario explored.

Figure 5. Scenario Results Table 29. Scenario analyses

Input Value

SROI

Notes

Lower value

0%

4.46

A lower value means less of the outcomes are attributable to other programmes.

Base case

25%

3.34

-

Upper value

50%

2.23

A higher value means more of the outcomes are attributable to other programmes.

5

4.49

Maximum duration for school-related outcomes are capped at the number of years the participant has left at school (based on mean age).

Variable

3.34

Outcomes last for a duration of 2-4 years in the base case.

1.92

Reducing the duration to 1 year across all outcomes exclusively assesses the short-term benefits of SROI, assuming net benefits are not maintained beyond this period, making it a more conservative scenario.

0%

4.15

Reducing the drop-off to 0% means that the first-year value of the outcome is maintained in subsequent years, and therefore, is less conservative.

Variable

3.34

Outcomes range from 4-58% in the base case, however the majority of outcomes have a drop-off of 50%.

3.26

A higher value is more conservative, as it increases the annual drop-off, and therefore reduces the proportion of benefit realised by the outcome in subsequent years.

0%

3.47

A lower value is less conservative, as it assumes that all benefits realised in the future are valued equally to those incurred immediately.

3.5%

3.34

-

Attribution

Duration (years)

Upper value

Base case

Lower value

1

Drop-off Lower value

Base case

Upper value

50%

Discounting Lower value Base case

Copyright Š Costello Medical Consulting Ltd

34

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

Upper value

5%

3.30

A higher value is more conservative, as it applies a higher discount to benefits realised in the future.

Sensitivity analysis A one-way sensitivity analysis was conducted on the base case results, to identify the inputs which the SROI was most sensitive to. Inputs used to calculate each outcome were varied one at a time, with an upper and lower value, and the impact on the SROI ratio recorded. The top ten variables that had the greatest impact on the SROI were recorded and shown on a tornado plot, as shown in Figure 6.

Copyright Š Costello Medical Consulting Ltd

35

Figure 6. One-way sensitivity analysis results

Future Recommendations In the absence of real data for several model parameter values, proxies were used to estimate outcomes relating to the behaviours and wellbeing of Noise Solution participants. In the future, to increase model precision and obtain more informed inputs, the proxies could be replaced with data collected directly from Noise Solution stakeholders, such as participants and their parents/guardians. In the tables detailing model inputs (see the Outcomes and Evidence section), these data have been highlighted orange. A summary of data recommended for collection, as well as the suggested means of doing so, have been detailed in Table 30.

To avoid recall bias, data regarding time periods prior to participation with Noise Solution should be collected before or on first contact with Noise Solution. Data relating to periods following Noise Solution should be collected after sufficient time has elapsed since finishing the programme for the relevant outcome to have been met (e.g. 3-6 months for short-term outcomes, 6-12 months for medium-term, and >1 year for sustained outcomes). The measurement unit should also be considered. For example, data relating to annual rates should, if possible, be collected one year after participation.

In a small minority of cases, where no feasible alternative pathways have been identified, participants of the Noise Solution programme may try and obtain funding to repeat the programme. For participants of Noise Solution who have taken part in the programme multiple times, data relating to the second participation should not be pooled with first-time programme participants. This is because, for second-use participants, data relating to the period before their second participation will be influenced by also including the period after their first participation. These data will therefore not truly reflect outcomes seen without Noise Solutionâ&#x20AC;&#x2122;s intervention and may underestimate the programmeâ&#x20AC;&#x2122;s benefit, as participants will be entering the programme with increased wellbeing compared to those with no prior participation. In the future, if the majority of participants are able to repeat the programme, these may also be incorporated.

Table 30: Future recommendations for data collection

Model input

Percentage of schoolaged participants with truant behaviour, pre and post Noise Solution

Related outcome(s)

Participants become reengaged with society through education, training or employment resulting in a reduced burden on government social services

Increased earnings resulting from reduced absences at school, and higher educational achievement

Average number of weeks non-school aged participants spent NEET in past 12 months, pre and post Noise Solution

Annual use of health services (GP, A&E, inpatient stay, outpatient/day case, any medication) for

Recommendations for collection This information could be collected in the form of a questionnaire, given both pre- and postparticipation in the programme. As education outcomes occur in the medium-term, data for post Noise Solution truancy should be collected >6 months after finishing the programme. If collection at this timepoint is likely to be biased by seasonality of the school schedule (such as summer holidays and study leave), then further follow-up is advised, or alternatively historical approximations may be used. For participants who are of school age upon entering the programme but not on finishing, data regarding truancy can still be collected for the period prior to programme participation.

To define truant behaviour, it should be asked if the participant did not turn up to school in a relevant time period, without authorisation. Severity of truant behaviour can be subjective, however to align with the scale in the Mental Health of Children and Young People in England study, it is recommended that a participant is marked as truant if they were truant once or more.10 Truancy as defined by missing individual lessons only should not be used, as this is often not picked up in school attendance data and therefore may be under-reported.31 To increase precision of data, it is recommended that parents/guardians of participants or school contacts be used to identify whether the participant has shown truant behaviour, rather than the participant themselves. However, in situations where bias may be introduced by parents/guardians condoning truancy, school contacts should be used where possible.

Participants become reengaged with society through education, training or employment resulting in a reduced burden on government social services

For participants â&#x2030;Ľ19 years prior to starting Noise Solution (and thus having been non-schoolaged prior to programme commencement), information on time spent NEET both pre- and post-participation should be collected. For participants who are school-aged upon starting Noise Solution but become non-school-aged shortly after finishing the programme, information on time spend NEET following programme participation should be collected.

To collect these data, a questionnaire could be issued both prior to and following programme participation. As the outcome is represented using a yearly average, the follow-up questionnaire would ideally be issued in the long-term (>1 year). However, in the absence of long-term data, shorter-term data (6 monthsâ&#x20AC;&#x201C;1 year) may be scaled to estimate an annual value, although this is likely to decrease precision. Demographics expected to influence this outcome, such as region (city vs rural), may also be collected, to select a more relevant dataset for use as a model input.

Wellbeing of participants improves, resulting in improved physical and mental wellbeing

As medical records are likely to be unobtainable, and resource use is across a broad range of services (meaning specialist health professionals are unlikely to give accurate data for unrelated services), this information may be most appropriately collected using parent/guardian or participant questionnaires before and after Noise Solution

Noise Solution – Social Return on Investment Report April 2019

behaviour-related concerns, pre and post Noise Solution

Proportion of schoolaged participants with behaviour-related complaints at school, pre and post Noise Solution

(parent/guardian questionnaire for participants aged <18 years, participant questionnaire for ≥18 years).

As the indicator is regarding annual use, the follow-up questionnaire should be issued at >1 year after finishing Noise Solution’s programme. Scaling short-term (<1 year) data to reflect annual use may be misleading for this outcome, due to natural fluctuations expected across the course of a year (for example, seasonal depression), but can still be used as a proxy in the absence of long-term data. The questionnaire should specifically aim to identify the use of behaviour concerns-related healthcare use (not all healthcare use).

Participant engagement in education increases, reducing school conduct problems

This information could be collected in the form of a questionnaire, given both pre- and postparticipation in the programme. As education outcomes occur in the medium term, data for complaints post Noise Solution should be collected >6 months after finishing the programme.

If collection at this timepoint is likely to be biased by seasonality of the school schedule (such as summer holidays and study leave), then further follow-up is advised, or alternatively estimates may be used. For participants who are of school-age upon entering the programme but not on finishing, data should still be collected for the period prior to programme participation.

To define this outcome, it should be asked how many times the participant received a formal complaint in the past 6 months. As this data is inputted as dichotomous, the participant should be defined as receiving behaviour-related complaints if this number exceeds a relevant threshold.

To increase precision of data, it is recommended that parents/guardians or school contacts (where possible) be used to identify whether the participant has received behaviour-related concerns at school, rather than the participant themselves.

This information could be collected using a parent/guardian questionnaire at three time points: • On/before first contact with Noise Solution (prior to)

Average number of days parents/guardians spend absent from work per year, prior to, during, and after Noise Solution

Increased labour productivity as parents/guardians take reduced time off work

Proportion of participants receiving

Reduced burden on the justice department and police services

Copyright © Costello Medical Consulting Ltd

•

On finishing Noise Solution (during)

•

>1 year after Noise Solution (after)

As this outcome is measured on an annual scale, the follow-up questionnaire would ideally be issued in the long-term (>1 year). Scaling short-term (<1 year) data to reflect annual absence may be misleading, as parental absenteeism is expected to be correlated with participant behaviour, which may naturally fluctuate across the course of a year (for example, seasonal depression). However, in the absence of long-term data, short-term data (<1 year) can still be used as a proxy.

This information could be collected in the form of a questionnaire, given both pre- and postparticipation in the programme. As the input regards an annual number, ideally the data 39

Noise Solution – Social Return on Investment Report April 2019

cautions/convictions, and annual number of these pre and post Noise Solution

Average educational resource use (educational psychologist, special needs support in classroom and classroom learning assistance) pre and post Noise Solution

Proportion of parents/guardians with increased stress as a result of child’s behaviour, pre and post Noise Solution

Proportion of siblings disrupted by child’s behaviour, pre and post Noise Solution

regarding cautions/convictions received following Noise Solution would be collected >1 year after finishing the programme. However, in the absence of long-term follow-up it is possible that shorter term data (6 months – 1 year) may be used to estimate an annual value.

To reduce the risk of bias, it is recommended that, where possible, parents/guardians are used to collect these data for participants aged <18 years, rather than the participant themselves. For participants aged ≥18 years, where parents/guardians may no longer be involved or aware of cautions/convictions, the participant themselves should be used to collect this data.

Reduced need for additional support in education and other services

This information could be collected in the form of a questionnaire, given both pre- and postparticipation in the programme. As education outcomes occur in the medium-term, data for complaints post-Noise Solution should be collected >6 months after finishing the programme.

When calculating the average resource use, time not spent at school should also be accounted for (e.g. school holidays, prolonged absences due to illness/behaviour).

For participants who are of school-age upon entering the programme but not on finishing, data should still be collected for the period prior to programme participation.

It should be asked if the participant has contacted each specific service (yes/no), and the frequency (for ease of input for the model, data should be collected as average annual use in hours, for each educational resource).

To increase precision of data, it is recommended that parents/guardians or school contacts (where possible) be used to identify average educational resource use, rather than the participant themselves.

Reduced stress and burden for family members/guardians

As the outcome occurs in the short-term, data regarding the post Noise Solution period should be collected 3-6 months after finishing the programme. For both pre and post Noise Solution, this information may be collected using a questionnaire. It should collect information on the number of parents/guardians, and how many of these experience increased stress levels as a result of their child’s behaviour. The Perceived Stress Scale (PSS) questionnaire may be employed as a tool to assess stress, although a simple yes/no response may also suffice.32 Due to the subjective nature of the data, it is recommended that the parents/guardians themselves be used for collection where possible.

Reduced stress and burden for family members/guardians

The data may be collected using a questionnaire issued before and after programme participation. As the relevant occurs in the short-term, data regarding the post Noise Solution period should be collected 3-6 months after finishing the programme. The questionnaire should enquire into the number of siblings, and how many of these are disrupted by the participant’s behaviour (yes/no).

Copyright © Costello Medical Consulting Ltd

40

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

Due to the subjective nature of the data, it is recommended that the siblings themselves be used for collection where possible. If this is not possible, and for siblings <6 years, it is recommended that parents/guardians, or alternatively school contacts, are used.

Proportion of parents/guardians who report increased connectedness to child

Increased family connections and relatedness

Copyright Š Costello Medical Consulting Ltd

As the outcome occurs in the short-term, data regarding the post Noise Solution period should be collected 3-6 months after finishing the programme, and may be done so using a questionnaire. It should collect information on the number of parents/guardians, and how many of these have experienced increased connectedness to their child (yes/no). It may also be beneficial to investigate this outcome using validated questionnaires which use a numerical metric to measure connectedness.

Due to the subjective nature of the data, it is recommended that the parents/guardians themselves be used for collection where possible.

41

References 1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.

Glenister S. A study of stakeholder perceptions of Noise Solution's practices: measuring impact on the well-being of youth facing challenging circumstances. Faculty of Education: University of Cambridge, 2017.

Boehm JK, Kubzansky LD. The heart's content: the association between positive psychological wellbeing and cardiovascular health.

Kern ML, Lyubomirsky S. Health benefits: Meta-analytically determining the impact of well-being on objective health outcomes AU - Howell, Ryan T. Health Psychology Review 2007;1:83-136.

Veenhoven R. Healthy happiness: effects of happiness on physical health and the consequences for preventive health care. Journal of Happiness Studies 2008;9:449-469.

Gutman LM, Vorhaus J. The Impact of Pupil Behaviour and Wellbeing on Educational Outcomes, 2012.

Jeremy Nicholls EL, Eva Neitzert, Tim Goodspeed. A Guide to Social Return on Investment. In: Network TS, ed, 2012.

Meltzer H, Gatward R. The mental health of children and adolescents in Great Britan. In: Statistics OfN, ed, 1999.

Green H, McGinnity A, Meltzer H, et al. The mental health of children and adolescents in Great Britain.

In: ONS, ed, 2004.

Collinson D. The mental health of children and young people in England. In: National Centre of Social Research, ONS, eds, 2017.

Department for Children Schools and Families. Youth Cohort Study & Longitudinal Study of Young People in England. 2007.

Department for Education. Youth Cohort Study & Longitudinal Study of Young People in England.

2009.

Department for Education. Post-16 Education: highest level achievement by age 25. 2018.

Tennant R, Hiller L, Fishwick R, et al. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health and Quality of Life Outcomes 2007;5:63.

Stansfield JC, B.; Timpson, H.; Whelan, G. Using and analysing WEMWBS to measure the impact of interventions in improving mental wellbeing. Cheshire and Merseyside Public Health Collaborative 2013.

Curtis L, Burns A. Unit costs of health and social care. In: PSSRU, ed, 2018.

ONS. Families with dependent children by number of children, UK, 1996 to 2017. 2018.

ONS. Families and Households. 2017.

ONS. Earnings and working hours, retrieved from: https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/earningsandworkinghours.

2019.

Romeo R, Knapp M, Scott S. Economic cost of severe antisocial behaviour in children - and who pays it. British Journal of Psychiatry 2006;188:547-553.

Education Endowment Foundation. Teaching and learning toolkit. 2018.

Department for Work and Pensions. Jobseeker's Allowance, retrieved from: https://www.gov.uk/jobseekers-allowance. 2018.

HM Treasury. The Green Book: Central Government Guidance on Appraisal and Evaluation. 2018.

McCrone P, Dhanasiri S, Patel A, et al. Paying the price - the cost of mental health care in Englandto 2026. 2008.

Ministry of Justice. Youth Justice Statistics 2016/17 England and Wales. 2018.

ONS. Crime in England & Wales, year ending September 2018. 2018.

Home Office. The economic and social costs of crime, second edition. 2018.

Brookes M, Goodall E, Heady L. Misspent Youth - The Costs of Truancy and Exclusion. New Philanthropy Capital 2007.

Impetus. Youth Jobs Index. 2017.

Bradly J, Bolas C. Social return on investment of substance misuse work, Leicestershire Youth Offending Service. Retrieved from: http://www.lsr-online.org/uploads/sroi-substance-misuse-final.pdf.

2013.

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

31.

32.

Attwood G, Croll P. Truancy in secondary school pupils: prevalence, trajectories and pupil perspectives. Research Papers in Education 2006;21:467-484.

Cohen S Fau - Kamarck T, Kamarck T Fau - Mermelstein R, Mermelstein R. A global measure of perceived stress.

Copyright Š Costello Medical Consulting Ltd

43

Noise Solution – Social Return on Investment Report April 2019

Appendix 1: Participant Interviews Noise Solution Participant Objective of the Questionnaire Costello Medical is assisting Noise Solution to define the value of the services they provide to help young people facing challenging circumstances. In order to do this, we are developing a social return on investment (SROI) model. SROI is a tool for measuring a broader concept of value, focussing on value itself rather than just money. For example, considering the additional social and/or environmental benefits of a service (e.g. increased opportunities for education, or reduced community crime), that extend beyond the direct economic benefits (e.g. increased employment and wages).

To develop a SROI that is truly reflective of Noise Solution, it is important to capture the outcomes experienced by various stakeholder groups (such as yourself) that are involved in or affected by Noise Solution’s services. The aim of this interview is to understand your experience with Noise Solution, so that we can make sure the SROI model reflects appropriate outcomes.

It is very important that you answer the questions honestly. Your answers are confidential and will not be shared in a way that is recognisable with Noise Solution, any of its staff, or your family/referring professional.

However, if there is constructive feedback you would like to share to be passed onto Noise Solution, we would be happy to do so. Please note that you do not have to answer every question, should you not wish to.

Background Information about You These questions help provide us with some important information about you, and how you came to participate in Noise Solution.

1. Before you began participating in Noise Solution, can you provide us with a bit of a background on how you spent your time, day to day?

For example, were you involved in any activities or hobbies that interested you? What did you enjoy doing?

2. When you were referred to Noise Solution, what (if anything) made you interested in participating with Noise Solution?

Your Participation in Noise Solution These questions relate more specifically to your involvement with Noise Solution, both during the weekly sessions, but also more broadly to how you think they have affected your life outside of the sessions (either positively or negatively).

3. How do you think participating in Noise Solution has had the greatest impact on you?

4. Have you noticed any changes in your life ‘at home’, with family or with friends, since participating in Noise Solution?

5. What role do you think the musician(s) and/or tutor(s) from Noise Solution play in your sessions? How would you say your relationship with them has changed over time?

6. Have you become interested in any more activities or learning (either within music tech or elsewhere) since participating in Noise Solution?

7. Since participating in Noise Solution, do you think your outlook for the future has changed?

8. How do you imagine your life would look now, if you didn’t have the opportunity to participate in Noise Solution?

Copyright © Costello Medical Consulting Ltd

44

Noise Solution – Social Return on Investment Report April 2019

Parent of Noise Solution Participant Objective of the Interview Costello Medical has been engaged by Noise Solution, to assist them in defining the value of the services they provide in helping people who face challenging circumstances. In order to do this, we are developing a social return on investment (SROI) model. SROI is a tool for measuring and accounting for the broader concept of value, focussing on value itself rather than just money. For example, considering the additional social and/or environmental benefits of a service (e.g. increased opportunities for education, or reduced community crime), that extend beyond the direct economic benefits (e.g. increased employment and wages).

To develop a robust, evidence-based SROI, it is important to capture the outcomes experienced by various stakeholder groups that are involved in or affected by Noise Solution’s services. The aim of this interview is to understand your experience, and your son’s experience, with Noise Solution, to appropriately reflect the outcomes that should be captured in the model.

Your answers are confidential and will not be shared in a way that is recognisable with Noise Solution, or any of its staff. However, if there is constructive feedback you would like to share to be passed onto Noise Solution, we would be happy to do so. Please note that you do not have to answer every question, should you not wish to.

Background Information of the Participant These questions help provide us with a general understanding of your child and how their engagement with Noise Solution was initiated.

1.

We are aware that your child has previously engaged with some of the services offered by the Norfolk and Suffolk NHS Foundation Trust. What services did (or does) they engage with through the NHS Foundation Trust?

2.

Before being referred to Noise Solution, were there any other services your child was engaged with or previously participated in? Were these kept ongoing alongside Noise Solution, or discontinued? What were the reasonings for doing so?

3.

What have been identified, either from yourself or others, as the key difficulties your child was experiencing which resulted in their referral to Noise Solution?

4.

Had you heard about Noise Solution prior to your child’s referral? Were you involved in the decision process of your child being referred?

Your Involvement with the Participant These questions relate to any observations or perceptions of your child’s behaviour and characteristics before and after his engagement with Noise Solution. These questions will help us get a better understanding on the type of changes that Noise Solution may or may not be facilitating.

5.

How well did your child engage in hobbies or activities outside of immediate school work before starting the programme, and how, if at all, has this changed once he began participating in Noise Solution?

Note: Aspects to consider may be his participation in extracurricular activities, such as sport or music, either associated with his school or within the community.

6.

How well did your child engage with education before participating in Noise Solution (formal education or otherwise), and how, if at all, has this changed once they began the programme?

Note: Aspects to consider may be their attendance (willingness to attend and/or actual attendance), relationships with peers, willingness to learn, academic performance, and engagement in group work in the classroom.

7.

Since your child’s involvement with Noise Solution, have there been any recognisable changes in their willingness to engage with other services, or the effectiveness of these services?

Copyright © Costello Medical Consulting Ltd

45

Noise Solution – Social Return on Investment Report April 2019

8.

How would you describe your child’s motivation to try new things or re-engage with previous hobbies, before and after their participation with Noise Solution?

9.

What behavioural or mood changes have you noticed from your child within their home environment, since participating in Noise Solution?

10.

Have your perceptions of, or relatedness with, your child changed since they started working with Noise Solution? Do you think these changes, if any, have been shared by other family members?

11.

For how long do you believe the effects of participating with Noise Solution’s programme to be?

12.

Given the previous questions we have just discussed, what have you noticed to be the most significant changes since your child began Noise Solution?

Your Experience of Noise Solution These questions relate to your involvement with Noise Solution, to try and understand how Noise Solution may have affected you and/or your family members (beyond your child). We would also like to understand your thoughts on Noise Solution’s role as a service to improve wellbeing, and how it differs from alternative services.

13.

Could you describe the changes in your life (and other family members’ lives) as a care giver, financially, emotionally, socially or professionally, since your child’s involvement with Noise Solution?

14.

How do you think the work done by Noise Solution differs from other services that your child has engaged in, both for the participants and their families?

15.

Do you have any other comments/thoughts that you would like to add? For example, are there any aspects you think could be altered or improved to be more beneficial for your child, and others participating in the programme?

Copyright © Costello Medical Consulting Ltd

46

Noise Solution – Social Return on Investment Report April 2019

Tutor of Noise Solution Participant Objective of the Interview Costello Medical has been engaged by Noise Solution, to assist them in defining the value of the services they provide to help people facing challenging circumstances. In order to do this, we are developing a social return on investment (SROI) model. SROI is a tool for measuring and accounting for the broader concept of value, focussing on value itself rather than just money. For example, considering the additional social and/or environmental benefits of a service (e.g. increased opportunities for education, or reduced community crime), that extend beyond the direct economic benefits (e.g. increased employment and wages).

To develop a robust, evidence-based SROI, it is important to capture the outcomes experienced by various stakeholder groups that are involved in or affected by Noise Solution’s services. The aim of this interview is to understand your experience with Noise Solution to appropriately reflect the outcomes that should be captured in the model. Some questions may also be asked regarding any changes you have noted in the participant (and other participants) since their involvement with Noise Solution.

Your answers are confidential and will not be shared in a way that is recognisable with Noise Solution, or any of its staff. However, if there is constructive feedback you would like to share to be passed onto Noise Solution, we would be happy to do so. You do not have to answer every question, should you not wish to.

Background Information These questions help provide us with a general understanding of your engagement with both Noise Solution, and specifically, your engagement with your tutee.

1.

We understand that you are currently working with your tutee as part of their participation in Noise Solution. Is your tutee the first participant you have worked with, and if not, how many participants have you worked with previously (or concurrently)?

2.

When did you first start working with Noise Solution, and what has your engagement been with Noise Solution since then?

3.

How did you hear about Noise Solution and become involved as one of the tutors?

4.

How long has your tutee been participating in Noise Solution with you for?

Noise Solution for Participants These questions relate to any observations or perceptions of your tutee’s behaviour and characteristics before and after their engagement with Noise Solution. These questions will help us get a better understanding on the type of changes that Noise Solution may or may not be facilitating.

5.

Since your tutee’s involvement with Noise Solution, have there been any recognisable changes in their willingness to engage with you, or your service?

6.

What behavioural or mood changes have you noticed from your tutee, since participating in Noise Solution?

7.

Have your perceptions of, or relatedness with, your tutee changed since their engagement with Noise Solution? Do you think these changes, if any, have been shared by other people in your tutee’s social network, such as family and friends that they share their blog with?

8.

What do you recognise as the most beneficial aspect of Noise Solution, both to your tutee directly as well as their family and broader community?

9.

For how long do you believe the effects of participating with Noise Solution’s programme to be?

Your Involvement with Noise Solution These questions relate to your involvement with Noise Solution, both as a service and how it impacts you directly as an employee.

10.

Beyond your work with Noise Solution, what other work do you engage in?

11.

Has your level of involvement changed over time, in either direction (increased or decreased)?

Copyright © Costello Medical Consulting Ltd

47

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

12.

What do you recognise as the most beneficial aspects of Noise Solution for participating musicians, such as yourself (either from an employment perspective, or more broadly)?

13.

What opportunities has Noise Solution provided you with, as a professional?

14.

What opportunities or interests, if any, has participating in Noise Solution led you into?

15.

Has Noise Solution had any effect on either the outlook of yourself, or the participants you work with?

16.

What, if any, aspects of Noise Solution do you think could be altered or improved from the perspective of musicians who work with the programme, such as yourself?

Copyright Š Costello Medical Consulting Ltd

48

Noise Solution – Social Return on Investment Report April 2019

Appendix 2: Excluded Outcomes from SROI Model Some outcomes that were identified as outcomes from Noise Solution and included in the Impact Map (see Figure 1) were not subsequently included in the SROI model calculations. These outcomes, and the respective justifications for exclusion, are summarised in Table 31.

Table 31. Impact map outcomes excluded from SROI Model

Period of Outcome

Short

Medium

Long

Outcome

Reason for Exclusion

Developing sense of competency in new area

Outcome implicitly include via associated medium-term outcomes, which are included in the model

Increased confidence, autonomy, sense of ownership and achievement

Outcome implicitly include via associated medium-term outcomes, which are included in the model

Increased engagement in pro-social activities and behaviours

Outcome implicitly include via associated medium-term outcomes, which are included in the model

Support of local community through musicians’ employment and studio hire

Stakeholder excluded from final scope of model

Alternative health/social services have increased awareness of Noise Solution

Outcome not expected to be associated with material value

Re-engagement with activities, education, employment or training

Outcome implicitly include via associated long-term outcomes, which are included in the model

Improved wellbeing

Outcome implicitly include via associated long-term outcomes, which are included in the model

Increased social capital

Outcome not expected to be associated with material value

Increased willingness to engage with other services

Outcome not expected to be associated with material value

Fulfilment for musicians resulting in improved wellbeing

Stakeholder excluded from final scope of model

Improved effectiveness of other health/social services resulting Noise Solution acting as a catalyst for change

Difficult to justify direct attribution to Noise Solution, without long-term data

Increased willingness from referral services to engage with Noise Solution

Outcome not expected to be associated with material value

Avoided knock-on emotional effects to siblings

Difficult to justify direct attribution to Noise Solution, without long-term data

Lower financial burden for family

Difficult to justify direct attribution to Noise Solution, without long-term data

Increased income tax from increased employment of musicians

Stakeholder excluded from final scope of model

Reduced burden on government-funded health services as referrals are redirected

Outcome not expected to be associated with material value

Copyright © Costello Medical Consulting Ltd

49

Noise Solution â&#x20AC;&#x201C; Social Return on Investment Report April 2019

and intervention limits escalation of problems

Copyright Š Costello Medical Consulting Ltd

50

Noise Solution – Social Return on Investment Report April 2019

Appendix 3: Academic Level Descriptions Based on the definitions applied in the Post-16 Education bulletin, Table 32provides a description of each academic level in the UK, that is applied in the model.

Table 32. Description of academic levels

Academic Level

Description

< Level 2

Below GCSE, or GCSE grades D-G

Level 2

GCSE grades A*-C or equivalent

Level 3

A-Levels or equivalent

Level 4

Grouped in the model; sub-degree higher level education such as a higher national diploma, or higher national certificate

Level 5 Level 6

Degree level higher education, such as bachelor’s degree with or without honours

Level 7

Post-graduate higher level-education such as master’s degree

Copyright © Costello Medical Consulting Ltd

51