Mapping the impact of MSF research: an evaluation framework and preliminary results Adam
1 Kamenetzky , 1Médecins
Background “MSF’s research should improve patients’ lives and the effective delivery of medical interventions. Does it?” MSF’s research is rooted in its experience in delivering humanitarian medical aid in resourcelimited and conflict settings.
Louise
Sarah
1 Venis ,
Rafael Van den
2 Bergh
Sans Frontières (MSF), London, UK; 2MSF, Brussels, Belgium
Methods
Results
Conclusions
We used a mixed methods approach. Quantitative information was sought via a survey, using a series of pre-defined impact indicators (A). Interviews with researchers explored perceptions of their experiences, and drew out underlying narratives (B).
Almost all respondents to the survey indicated their work had been disseminated. Most also reported contribution to policy change.
The workload associated with delivering the questionnaire survey method was not felt to be sustainable.
Few indicated their research had directly improved MSF programmes or patient outcomes.
The range of research methods and topics within MSF necessitates a more nuanced approach.
A. Survey (indicators described by Zachariah et al.)2
In the past 20 years, MSF has published over 1,000 articles in peer-reviewed journals.1 Yet the contribution of MSF research to real-world outcomes (e.g. programme and policy change) and impacts (e.g. improvements in patient care) has not been systematically analysed across the organisation.
1 Bishop ,
% of projects responding to survey (n=27) reporting achievements against indicators
Dissemination • Publication • Interviews/workshops • Presentations/media
n=31* projects *approved by MSF/ national ethical review boards in 2009
principal investigators emailed survey; self-reporting
Attempts to map the longterm impact of MSF’s research must use a comprehensive series of indicators.
100 90
Policy/practice
Both quantitative (metric) and qualitative (narrative) information is needed to build a complete picture of MSF’s research achievements.
80
• Rapid advice/circular • Guidelines • Training materials
70 60 50
B. Interviews (topic guide informed by focus group)
40 30
Operations
Efforts are underway to evaluate whether a standardised series of outcome measures can be used to document MSF’s research impact prospectively across a cohort of MSF researchers.
20 10
Impact?
Publication n=10* interviewees purposively selected; *at end April 2013
principal investigators telephoned; semi-structured interview
0
interviews transcribed; themes identified through an iterative and deductive process
Met one or more criteria Contributed to Improved national / for performance of dissemination international MSF policy programmes
Policy
Left: snapshot of categories of questions forming the basis of a standardised set of medical research impact evaluation criteria, currently being trialled in MSF via a pilot of Researchfish – a system adopted by a majority of UK medical research funders.
Iterative analysis of interview transcripts is currently underway.
Advocacy Above: efforts to map the impact of MSF’s research must take into account the complex network of associations that link different areas of its humanitarian work.
Preliminary findings suggest: •
Recently, MSF has defined a series of indicators to assess the impact of its operational research.2 And UK charitable funders have joined together to develop a practical framework with which to evaluate medical research impact.3
We conducted a pilot study with the aim of determining how best to capture and evaluate the long-term impact of MSF research. Here we present preliminary findings.
Reduced patient morbidity / mortality
•
• Above: stuck in the mud? Innovation in how MSF shares research information within and outside of the organisation could prevent valuable data becoming internally siloed, or lost when field staff move on (photo w/ permission from Jen Jennings).
The importance and relevance of research to patient care is not well understood in many MSF project settings. There is uncertainty about what constitutes “good” research, and the value of collecting data in a consistent way to improve MSF’s operations. Documenting the real-world outcomes of research as they happen is crucial for realistic and thorough impact reporting and evaluation.
References 1
Bishop, L et al. Trends in MSF research publications. Poster: MSF UK Scientific Day. 10 May 2013 2
Zachariah R et al. Is operational research delivering the goods? The journey to success in low-income countries. Lancet Infect Dis. 2012 May;12(5):415-21 3 Association
of Medical Research Charities: Tracking the impact of charity research funding. 13 February 2013, available at: bit.ly/10Oz8Xv