Community Outcomes Roundtable - 01/05/14

a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk

|

tw: @GenChangeUK| t: 0207 014 2690

Community outcomes roundtable – write up When: Thursday 1st May 2014 Where: vInspired, Dean Bradley House, Horseferry Road, London Attendance: Chair:

Hannah Mitchell

Head of Knowledge

vInspired

Paul Adnitt

Curriculum Manager

NCS Trust

Alice Thornton

Head of Impact and Policy

Student Hubs

Xenis Davis

Programme Manager

Envision

Nick Ockenden

Head of IVR

NCVO

Jon Adamson

Senior Research Manager

CFE

Ewan King

Director

OPM

Amy Skipp

Research Director

NatCen

Rania Marandos

Deputy CEO

Step Up To Serve

Background In June 2013, the Young Foundation published a Cabinet Office commissioned scoping paper setting out a Quality Framework for youth social action.1 The framework includes a definition, set of principles and an outcomes framework that young people would build up on their social action journey from aged 10 -20. Generation Change originally set out the brief

for commissioning this framework and was instrumental in setting out the “double benefit” language as a guiding model for quality youth social action.

In December 2013, Generation Change was then commissioned by the Cabinet Office to expand upon the “individual benefits ” part of the double impact framework by developing a

1

Slide 11, “Scoping a Quality Framework for Youth Social Action”, The Young Foundation, June 2013

common metric for organisations to assess outcomes against each of the 8 skills areas.

Generation Change brought together stakeholders from multiple sectors and staff from member organisations to identify the challenges of reporting outcomes in this area. Our recommendations were supported by IVR and the Young Foundation, and set out clear options for developing reporting in this area.

Double benefit model

Generation Change has now been appointed by Step Up To Serve to lead its project on implementing a quality framework for youth social action programmes, taking forward these developments, and drawing on our delivery organisations. We have established a pilot group of professionals directly involved in evaluation activities in order to support the Step up to Serve campaign by developing common approaches that embed the quality principles and outcomes framework.

It is evident that the first step in this project is to expand upon how to evaluate and measure the “benefits to society�, which forms the other half of the double benefit framework. This was one of Generation Change, IVR and the Young Foundation’s recommendation to the Cabinet Office when developing a common metric. With different social action programmes having unique interventions into a range of social causes, it appears more difficult to assign a set of common outcomes which all youth social action programmes could be asked to report on.

On 1st May 2014, Generation Change hosted a roundtable with research agencies and evaluation specialists for an initial scoping session for this work. The session was chaired by Hannah Mitchell, Head of Knowledge at vInspired, and written up by David Reed, Forum Coordinator at Generation Change.

a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk

|

tw: @GenChangeUK| t: 0207 014 2690

Defining “community outcomes” A range of possible definitions were suggested at the roundtable:

Community outcomes are…

Any outcomes that are not a benefit to the individual. A negative definition, in that beneficial outcomes that result from a social action programme, not directly related to the individual taking part, is considered a “community outcome”.

Problems:

Some changes in the individual are considered socially beneficial

•

(e.g. community cohesion, reduction of violence etc) Longer-term impacts derive from behaviour change in the

•

individual; such as an increased propensity to give, being more socially minded in future decision making, etc.

Role modelling of positive behaviour – the fact that youth social

•

action is taking place at all, rather than not, could have community outcomes in itself or a knock on effect.

Defined by the indicator or cause. Most social action projects take a theory of change approach whereby the outcomes are determined by the analysis of a social problem / need.

Each cause will have its own set of indictors.

Problems:

•

Endless hair splitting of different causes – e.g. “environment” could be: “biodiversity, conservation, climate change, tidiness, community spaces…” etc.

•

This evaluation approach alone does not leave social action with a framework that is easily comparable.

Beneficiaries / communities that experience impact. Many programmes attempt to identify a beneficiary, or beneficiary community, and measure benefit directly there.

Problems:

•

Definition of community can be discrete based on the individual

•

Some interventions have positive impacts that are not easily tied to a specific beneficiary community (e.g. tree-planting)

•

Localities and networks play a significant role in understanding impact, complicating this picture of beneficiaries.

a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk

|

tw: @GenChangeUK| t: 0207 014 2690

Defined by the young person. Many providers – e.g. NCS, vInspired, Envision, Fixers frame the “community impact” part of their programme as up to the young person to decide in terms of the cause and means of addressing it. In this case, each project will have its own approach to defining community impact.

Problems:

•

Difficult to compare or contrast different evaluation approaches

•

Social action projects need to support young people to learn about community impact, so this definition does not solve the initial problem.

Conclusions of this section:

•

None of these definitions is adequate alone, or suitable in all cases

•

The following areas below appear important to separate: Outputs Positive change added

Reduction in a negative indicator

Impact of young people

Impact of youth social action generally

Immediate / direct impact Behavioural changes Micro impact

•

Outcomes

Cumulative / culture shift Attitudinal changes Macro impact

Whereas there seems to be importance in the inter-relation of the following:

E.g. mentoring, awareness

Mode of intervention

E.g. biodiversity, homelessness,

campaign, fundraising

educational disadvantage

Cause

Beneficiaries

E.g. elderly care home residents, bullied under 16s, my local park

Measuring “community outcomes” Kind of measure used

E.g.

Outputs

…Numbers taking part, attendance of events, tweets

Responses from direct beneficiaries

…Survey of elderly person, report cards, focus groups

Direct set of indicators around a particular issue

…Wellbeing metrics for mentoring of bullying

Universal positive indicators

…Employment rates / voter intention

Responses from immediate social circles

…Teachers / parents / facilitators / friends

Attitudinal statements and future beliefs

…“I believe I am more likely to get a job now”

Second party data / indicators

…attainment, indices of deprivation, local voter turnout

Infrastructural / economic benefits

…capacity of providers, transport links, economic benefits

Media and policy

…news agenda, endorsements, media coverage

A possible means of modelling these measurement approaches was identified: Outputs, direct outcomes and indicators, immediate responses Longer term impact

Close social circle / network

atitudinal statements and beliefs, positive specific indicators, second party data

Individual

Direct beneficiaries

Broader community

Wider / universal data, infrastructure and economics, media and policy

Conclusions of this section: •

Measuring the outcomes of a specific project is different to measuring the broader community outcomes of social action generally, although they are linked.

•

The former might require flexible frameworks for determining appropriate measures, whereas it might be possible to set out general outcomes for the latter (next section)

a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk

|

tw: @GenChangeUK| t: 0207 014 2690

Common long term outcomes Roundtable participants were asked what they would want to be able to measure in an ideal world to demonstrate the long-term community outcomes of social action taking place. The following suggestions made were not intended to be exhaustive: Measurement scale and focus “Individual”

“Communities”

“Society”

Positive traits more common

Responsible and engaged

Active citizenship

and recognised (e.g. social,

communities

ambitious, compassionate, engaged) Changes in behaviours (e.g.

Generational qualities –

giving, decisions, choices)

understanding / acceptance of

Cohesion, tolerance

social differences Increased social capital / life

Social problems alleviated

chances with positive multipliers

(e.g. greater wellbeing)

Voice and influence of young

Changed social attitudes

people

towards young people

Shift in social trends

Participation

Practical challenges Providers cannot be a subject specialist in every cause! Some programmes run a very specific intervention (e.g. mentoring in schools) that is easier to build an impact model around; whereas providers such as vInspired, Envision, NCS and Fixers effectively have multiple interventions, many of which might be up to the young person to decide upon in terms of the cause, means of addressing that cause, theory of change, etc.

Solution: Do Something uses standard evaluation frameworks based on the most common intervention approaches (e.g. “mentoring”, “awareness campaign” etc) Requirement for a control group. Proving outcomes puts demands on organisations to use sophisticated approaches such as RCTs which is not feasible to run for multiple ages, contexts and programme lengths in all cases.

Solution: Use derived data, external evidence bases, and / or national indicators to provide baseline control areas to compare to. Open data solutions could make it much easier to identify and respond to community outcomes.

a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk

|

tw: @GenChangeUK| t: 0207 014 2690

Attributing impact over a longitudinal period. Most of the meaningful outcomes organisations need to prove are not captured within the timeframe of a single project, but might be experienced long after a young person has left the programme.

Solution: similar to above solution, common data sets needed.

Actions •

Capture and compare different interpretive models for community outcomes for consideration by the group

•

Propose a definition of “active citizenship” to set out a vision for the kind of outcomes youth social action programmes seek

•

Look into existing template frameworks used to help clarify outputs and outcomes for different forms of social action (e.g. “mentoring, planting trees”)

•

Use a deliberative engagement exercise with OPM to capture young people’s own interpretation of the kinds of community impact they’re having and would like to have.

•

Compile and review existing evidence from member programmes in this area and compare to common standards of evidence produced by TSIP

•

Identify solutions with Step Up To Serve to national data indicators and insights that could help establish baseline figures.