Writing a REF Impact Case Study Impact of Social Sciences Project LSE Public Policy Group
The current state of play • 360 days to REF deadline • Survey of submission intentions due in tomorrow (only 300 words in total) • Universities have a first draft of prospective cases • But these still need quite a lot of work • How your institutions are handling things will vary • There is still time for action – but much uncertainty, and we have no magic or official answer
Overview of the session Today is a discursive session on key uncertainties: 1.
Uncertainties around reach and significance
Envisioning the overall narrative
Proving 2* quality for underpinning research
What is â€˜impactâ€™ and what is not?
Maximizing intensity of impact evidence
1. Uncertainties around reach and significance
Definitions of reach and significance â€˘ Reach is understood as extent and diversity of the communities, environments, individuals, organizations or any other beneficiaries that have benefitted or have been affected. â€˘ Significance is understood as the degree to which impact has enriched, influenced, informed or changed policies, opportunities, perspectives, or practices of communities, individuals or organizations.
Mapping HEFCEâ€™s impact criteria
Definitions of reach and significance • What is the relationship between the two? Can increased reach compensate for reduced significance? • Emphasis on showing change – this implies need to demonstrate linear effects • Uncertainty around things like generating debate in a profession, engagement over proposition not taken up • Is this biased towards public policy impacts? • Panels reassure us that iterative, dynamic, and narrative accounts will be expected and encouraged
2. Envisioning the overall narrative
Think creatively about the narrative • It’s all about the narrative – and the ‘so what?’ question • Start with the user side and work backwards • Not necessarily your most cited academic work – where have you engaged? • Research outputs must be ‘credibly linked’ to impact • Corporate versus individual submissions • Integrate different streams of work or expertise in the department
The case study narrative IDEALLYâ€Ś Short summary of the case (100 words)
Strong opening headlining the research and impacts
Underpinning research (500 words)
Narrative account of how the research evolved
References to the research (max. 6)
Majority of 2* references to research outputs
Details of the impact (750 words) Corroborating sources Max. 10 (5 individuals)
Building to a strong, factoid-intensive narrative on impacts Good mix of references and testimonies,
IN REALITYâ€Ś Imbalance towards research rather than impacts Over-emphasis on research outputs and weak quality indicators Excessive focus on dissemination and advice Lack of factoid evidence and weak signs of self-research, over-reliance on weak testimonials
Process trace your impact back to your underlying research Effects Changes
Impact of outputs or activities of external body/actors
Public engagement contributions
Intermediate impacts â€“ usage or cites, by external actors
Partnership funding / Direct contacts = auditable impacts Underpinning research Evidence of (strong) dissemination
Public engagement contributions
3. Proving 2* quality of underpinning research
Deciding what research to submit • All work must cross the 2* quality threshold (i.e. it must be ‘internationally recognized’) • Easier for traditional academic outputs (i.e. already in previous RAE submissions) • However impact research outputs may be more applied: e.g. grey literature • Panel will take a look, if not obviously 2*, then it is read. • Therefore need to reassure the Panel that this work is of the required quality via another means
Critically assessing your own research • Has any of your impact output been assessed e.g. through an internal or organisational review process? • For research reports, was an evaluation completed on outputs from a programme? • Panels C and D accept outputs that can be shown to be highly cited or form a ‘reference point for further research beyond the original institution’ • If none of these apply, then you or your Department will need to organise your own review or evaluation
4. What is impact and what is not?
What is feasible in assessing impact?
Writing about your impacts Net impacts
– Danger of over-claiming or making infeasible estimates – Making informed guesstimates at net effects
– Maximizing the level of detail and specifics at this level – always be above this threshold
– Dissemination activities on their own without any follow-on interaction or take up – Tricky in the area of public dissemination – a radio broadcast is not enough – Something that is generally part of academic service – Just providing expertise and advice to however important a person/organization
5. Maximizing intensity of impact evidence
Research your impact proactively • • • •
750 words max to demonstrate intensive impacts Narrative should not detract from specifics / data Being your own detective and research yourself Start with Google and see where your research has been picked up by external users • Then carefully document official and unofficial dissemination that you undertook • Outline as clearly as you can the ‘effect, change or benefit’ that your research made.
Research your impact proactively II • Can you be precise about who exactly used your work in business, government, or third sector? • Can you offer at least some good evidence of how they used it and the effects it had? • Did your contacts or involvement continue over a period of time? And did it had knock-on effects? • Did people pay for your work or advice? Can you track all the flows of money?
For more details see: Impact of Social Sciences blog: http://blogs.lse.ac.uk/impactofsocialsciences/ Email: email@example.com Twitter: @lseimpactblog Facebook: Impact of Social Sciences
Definition of impact • ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ • INCLUDES raising the level of public debate and public scrutiny, public engagement, changing policy and practices, use by professional bodies. • EXCLUDES (a) impact on research or the advancement of academic knowledge within the HE sector (b) acting as an expert or advisor to a public body
The REF Main Panels Main Panel A covers: Clinical Medicine Public Health, Health Services and Primary Care Allied Health Professions Psychology, Psychiatry and Neuroscience Biological, Agricultural and Food Sciences
Main Panel B covers: Earth Systems and Environmental Sciences Chemistry and Physics Mathematical Sciences Computer Science and Informatics All areas of Engineering
Main Panel C covers the social sciences including: Anthropology; Business and Management; Education; Economics; Geography, Law; Politics and IR; Social Policy; Sociology
Main Panel D covers the humanities including: Art and Design; Dance; Drama; Communication and Media Studies, English; History; Modern Languages; Music; Philosophy; Theology;
More on the social science REF panel • One user appointed per sub-panel • These appointees will coordinate recruiting additional research users (by Spring 2013) • Two readers of case studies – 1 user, 1 academic • Not ‘user-led’ but users will play a key role in assessment
HEFCE guidance on underpinning research
HEFCE guidance on recording impacts
HEFCE guidance on impact evidence
For public engagement cases You need to be able to answer these questions: • Can you show evidence of your research being strongly disseminated? • Did you undertake outreach activities? • Can you show evidence of your research being widely received? • Were there secondary or follow-on effects? i.e. reaching additional audiences, building new interactions, creating additional outputs.