How to: Writing a REF Impact Case Study

Writing a REF Impact Case Study Impact of Social Sciences Project LSE Public Policy Group

The current state of play • 360 days to REF deadline • Survey of submission intentions due in tomorrow (only 300 words in total) • Universities have a first draft of prospective cases • But these still need quite a lot of work • How your institutions are handling things will vary • There is still time for action – but much uncertainty, and we have no magic or official answer

Overview of the session Today is a discursive session on key uncertainties: 1.

Uncertainties around reach and significance

2.

Envisioning the overall narrative

3.

Proving 2* quality for underpinning research

4.

What is ‘impact’ and what is not?

5.

Maximizing intensity of impact evidence

1. Uncertainties around reach and significance

Definitions of reach and significance • Reach is understood as extent and diversity of the communities, environments, individuals, organizations or any other beneficiaries that have benefitted or have been affected. • Significance is understood as the degree to which impact has enriched, influenced, informed or changed policies, opportunities, perspectives, or practices of communities, individuals or organizations.

Mapping HEFCE’s impact criteria

Definitions of reach and significance • What is the relationship between the two? Can increased reach compensate for reduced significance?

• Emphasis on showing change – this implies need to demonstrate linear effects • Uncertainty around things like generating debate in a profession, engagement over proposition not taken up • Is this biased towards public policy impacts?

• Panels reassure us that iterative, dynamic, and narrative accounts will be expected and encouraged

2. Envisioning the overall narrative

Think creatively about the narrative • It’s all about the narrative – and the ‘so what?’ question • Start with the user side and work backwards • Not necessarily your most cited academic work – where have you engaged?

• Research outputs must be ‘credibly linked’ to impact • Corporate versus individual submissions • Integrate different streams of work or expertise in the department

The case study narrative IDEALLY‌ Short summary of the case (100 words)

Strong opening headlining the research and impacts

Underpinning research (500 words)

Narrative account of how the research evolved

References to the research (max. 6)

Majority of 2* references to research outputs

Details of the impact (750 words) Corroborating sources Max. 10 (5 individuals)

Building to a strong, factoid-intensive narrative on impacts Good mix of references and testimonies,

IN REALITY‌ Imbalance towards research rather than impacts Over-emphasis on research outputs and weak quality indicators Excessive focus on dissemination and advice Lack of factoid evidence and weak signs of self-research, over-reliance on weak testimonials

Process trace your impact back to your underlying research Effects Changes

Benefits

Impact of outputs or activities of external body/actors

Public engagement contributions

Intermediate impacts – usage or cites, by external actors

Partnership funding / Direct contacts = auditable impacts Underpinning research Evidence of (strong) dissemination

Public engagement contributions

3. Proving 2* quality of underpinning research

Deciding what research to submit • All work must cross the 2* quality threshold (i.e. it must be ‘internationally recognized’) • Easier for traditional academic outputs (i.e. already in previous RAE submissions) • However impact research outputs may be more applied: e.g. grey literature • Panel will take a look, if not obviously 2*, then it is read.

• Therefore need to reassure the Panel that this work is of the required quality via another means

Critically assessing your own research • Has any of your impact output been assessed e.g. through an internal or organisational review process?

• For research reports, was an evaluation completed on outputs from a programme?

• Panels C and D accept outputs that can be shown to be highly cited or form a ‘reference point for further research beyond the original institution’ • If none of these apply, then you or your Department will need to organise your own review or evaluation

4. What is impact and what is not?

What is feasible in assessing impact?

Writing about your impacts Net impacts

– Danger of over-claiming or making infeasible estimates – Making informed guesstimates at net effects

Gross impacts

– Maximizing the level of detail and specifics at this level – always be above this threshold

Non-impacts

– Dissemination activities on their own without any follow-on interaction or take up – Tricky in the area of public dissemination – a radio broadcast is not enough – Something that is generally part of academic service – Just providing expertise and advice to however important a person/organization

5. Maximizing intensity of impact evidence

Research your impact proactively • • • •

750 words max to demonstrate intensive impacts Narrative should not detract from specifics / data Being your own detective and research yourself Start with Google and see where your research has been picked up by external users • Then carefully document official and unofficial dissemination that you undertook • Outline as clearly as you can the ‘effect, change or benefit’ that your research made.

Research your impact proactively II • Can you be precise about who exactly used your work in business, government, or third sector?

• Can you offer at least some good evidence of how they used it and the effects it had?

• Did your contacts or involvement continue over a period of time? And did it had knock-on effects?

• Did people pay for your work or advice? Can you track all the flows of money?

For more details see: Impact of Social Sciences blog: http://blogs.lse.ac.uk/impactofsocialsciences/ Email: impactofsocialsciences@lse.ac.uk Twitter: @lseimpactblog Facebook: Impact of Social Sciences

Supplementary Slides

Definition of impact • ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ • INCLUDES raising the level of public debate and public scrutiny, public engagement, changing policy and practices, use by professional bodies.

• EXCLUDES (a) impact on research or the advancement of academic knowledge within the HE sector (b) acting as an expert or advisor to a public body

The REF Main Panels Main Panel A covers: Clinical Medicine Public Health, Health Services and Primary Care Allied Health Professions Psychology, Psychiatry and Neuroscience Biological, Agricultural and Food Sciences

Main Panel B covers: Earth Systems and Environmental Sciences Chemistry and Physics Mathematical Sciences Computer Science and Informatics All areas of Engineering

Main Panel C covers the social sciences including: Anthropology; Business and Management; Education; Economics; Geography, Law; Politics and IR; Social Policy; Sociology

Main Panel D covers the humanities including: Art and Design; Dance; Drama; Communication and Media Studies, English; History; Modern Languages; Music; Philosophy; Theology;

More on the social science REF panel • One user appointed per sub-panel • These appointees will coordinate recruiting additional research users (by Spring 2013) • Two readers of case studies – 1 user, 1 academic • Not ‘user-led’ but users will play a key role in assessment

HEFCE guidance on underpinning research

HEFCE guidance on recording impacts

HEFCE guidance on impact evidence

For public engagement cases You need to be able to answer these questions: • Can you show evidence of your research being strongly disseminated?

• Did you undertake outreach activities?

• Can you show evidence of your research being widely received?

• Were there secondary or follow-on effects? i.e. reaching additional audiences, building new interactions, creating additional outputs.