12 minute read
Myths, mysteries and the art of reviewing evidence
Myths, mysteries and the art of reviewing evidence Reading evidence reviews during COVID-19 and beyond
By Rhian Noble-Jones
This year has been unique at every level.
Our fear and curiosity of the mystery virus, COVID-19, will remain in our memories for many reasons. News outlets and social media are trying to make sense of a flood of information. We are trying to separate an everchanging truth, from myths and ‘fake news’. The situation has highlighted, how vulnerable those with ‘underlying conditions’ can feel; the disparity of salaries (or ‘perceived worth’) between soccer players, celebrities and nurses, physiotherapists, clinical researchers; and of course, the compassion and resilience of our healthcare teams. However, the international collaboration in research during the pandemic has been wonderful to witness, as has the notion that political decisions should be ‘guided by the clinical evidence’. It has brought research vocabulary into the everyday language of clinicians and journalists alike. The astounding swiftness in ‘expedited ethical boards’, ‘accelerated assessments’ and ‘rapid reviews of the evidence’ has been, at times, awesome.
A particular product of the pandemic has been the Rapid Review. Over the course of 2-3 weeks there appeared rapid reviews of COVID-19 in relation to healthcare roles (e.g. critical nurses, respiratory physiotherapists), particular pathologies (e.g. asthma, diabetes) and particular contexts (e.g. Critical Care, Palliative Care). Rapid reviews are particularly popular with policy makers, as they are quick enough to respond to in situational or time sensitive issues. The process aims to be rigorous and transparent but in order to reduce the time taken, compared to a full systematic review, the focus may be limited in breadth or depth; or the team has to be very large (requiring greater funding). They may include a ‘review of reviews’ (see later) and sometimes curtail the review and synthesis stages 2 , hence increasing the risk of bias. For a lymphedema-based example, see a rapid review of compression wraps (Table 1).
This article aims to set the Rapid Review in relation to other types of evidence reviews and briefly show how different types of review have informed aspects of our lymphedema field.
Regardless of COVID-19, for anyone interested in lymphedema research, the last few years have been notable. The amount of research directly relating to lymphedema has
Dr. Rhian Noble-Jones, PhD, PgD, PgC, is a National Lymphoedema Researcher for Wales, Associate Lecturer for Swansea University Wales, vice-chair of the Scientific Committee of the British Lymphology Society and the Programme-lead for Lymphoedema at University of Glasgow, Scotland. grown each year. At the same time, there is increasing realization that related research— in genetics, adipose tissue, inflammation and pharmaceuticals—might completely change the way we understand and manage lymphedema. In this ocean of information there is, quite rightly, an expectation that health care professionals (and the academics who teach them) will remain aware of the latest evidence to support clinical practice. Patients and their families are becoming increasingly interested and aware of medical news announcements, through both traditional and social media streams. The COVID-19 pandemic seems to have heightened this expectation.
Despite an acknowledgment that medicine has always been a mixture of science and art 3 we like to think that in the 21 st century our understanding of underlying ‘facts’ is improving. The assumption of a review is that there is a depth and breadth of research to be analyzed. However, the reality is that there are two major issues influencing the amount of research in a medical field: funding and bias. It is difficult to access research funding to investigate many chronic pathological conditions. They are not as emotive or frightening as life-threatening disease and pharmaceutical or advanced technology companies sometimes struggle to see the investment potential. In relation to bias,
medical science has always had its biases (intentional and unintentional) but amplifying this bias is increased sophistication in the use of sound bites and online ‘baiting hooks’, which mislead people with fake news. We would all like to find a simple cure. Subconsciously, people search for confirming evidence, or listen to information that confirms existing practice rather than that which requires a change in behaviour. As busy people, we are very glad if someone else has reviewed the evidence for us and a review by others can seem more objective. So evidence reviews have become popular. However, how do we remain critical when an article is called ‘a review of evidence’?
There are many types of review. Each review type has its strengths and weaknesses, and appropriate purpose 6 . In essence, each one has four stages: Search, Appraise, Synthesize and Analyze 2 . The difference between each review type is how each of these stages is conducted; although, as Grant and Booth (2009) found, the boundaries are fuzzy. Nevertheless, understanding the process undertaken helps you to understand the level of rigour and therefore how much value to place on the review that you are considering in relation to your intended use.
The gold standard and probably the most well known type of review of evidence is the Systematic Review, such as conducted for a Cochrane Review (https://canada.cochrane. org/). It follows a very rigorous process for each stage 4 . It requires a pre-determined protocol and a team of researchers rather than a lone researcher/student. It seeks to draw together all known knowledge on a topic area. Traditionally it uses a hierarchy of studies where randomized controlled trials are considered supreme but recently a wider range of study types have been included. The disadvantage is that it takes a long time and can be costly. An example paper is given in relation to the breast-cancer related lymphedema (Table 1).
The most technically demanding of reviews is the Meta-analysis. There are very few of these in the field of lymphedema as its purpose is to combine, statistically, the results of several quantitative studies that are similar in design. A limitation is finding sufficiently similar studies. It can be very useful where there are many small studies with insufficient statistical power on their own to influence practice. An example in relation to the incidence of arm lymphedema is given (Table 1). The qualitative research equivalent would be a Qualitative Evidence Synthesis. In contrast to the summative nature of the previous, this process looks for themes/constructs across qualitative studies and attempts to give an overarching narrative or new theory.
Post-graduate students learn research skills by conducting a Systematized Review toward their PhD. This is a smaller version of the systematic review. The process mimics that of the systematic review but on a narrower or more limited scale, typically due to time or the number of people involved. An example is given in relation to inter-professional education (Table 1). A slightly simpler process again is the Literature Review that might be undertaken for a journal article or chapter
in a textbook. It looks for what is known or accomplished in a topic, sums up, and avoids duplication of research but it may be of limited breadth and therefore can hold bias. An example is given in relation to lymph node transplants (Table 1).
TABLE 1
Examples of types of review
Type of review
Systematic review
Systematized review
Literature review
Overview
Mapping review
Scoping review
Rapid review
State-of-the-art review
Critical review
Meta-analysis
Qualitative evidence synthesis
Mixed methods review/ Realist review
Example papers
https://doi.org/10.1002/14651858.CD011433.pub2 and http://epubs.surrey.ac.uk/842664/
https://www.ncbi.nlm.nih.gov/pubmed/29966884
https://journals.lww.com/annalsofsurgery/Fulltext/2015/05000/ Vascularized_Lymph_Node_Transfer_for_Treatment_of.31.aspx
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5497342/
http://eprints.gla.ac.uk/50066/1/50066.pdf
https://www.gov.scot/publications/smasac-short-life-workinggroup-lymphoedema-lymphoedema-care-scotland-achieving-equityquality/pages/5/
http://www.medidex.com/evidence-based-procurement-boardebpb/864-compression-wraps-for-lymphoedema-rapid-reviewfeb-2017.html
https://www.ncbi.nlm.nih.gov/pubmed/30248726
https://www.cancernurse.eu/documents/ EONSEuroPEPLymphoedemaCJONArticle.pdf
https://www.sciencedirect.com/science/article/pii/ S1470204513700767?via%3Dihub
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4153404/
https://njl-admin.nihr.ac.uk/document/download/2011415
A review which is entitled ‘an Overview of…’ takes a big picture approach. It usually describes the type of literature on a subject or gives a broad summation of a topic area. It can be useful for newcomers or to understand a different field e.g. surgical management of lymphedema (Table 1).
It is important to recognize that a review that is not conducted systematically, can only give an estimated (near truth/best guess) answer. Arguably, the further away from the Cochrane process, the more likely it is to reach biased or wrong conclusions. However, the rigour, timescale and investment required of a full systematic review is not always possible and may not be pragmatic for policymakers; nor are they always required. Alternative review processes attempt to balance the need for rigour and relevance, with the reality of funding and human resources. Hence the rise of the rapid review in the rushed context of COVID-19. A middle ground between the haste of rapid reviews for policy and the academically thorough systematic review is ‘middle-ground research’ 5 or Mixed-method/ Realist Review. This process attempts to bring together the ‘what works’ of quantitative evi dence with the ‘how and why it works’ of qualitative studies to give us a better understanding of ‘what works under what circumstances’. There is a great deal of interest in this field of research, particularly as we enter the era of Realistic Medicine 8 . Realist review assumes a high level of stakeholder involvement and an element of negotiation between reviewers and stakeholders 7 . Other reviews have specific purposes but can appear very similar to those above: n Mapping Review: Useful for identifying gaps in the evidence or setting a context. Used when little is known of what currently exists. Quick, broad descriptive picture; might oversimplify. n Scoping Review: Provides a quick first assessment of quantity and quality of literature on a subject; but is still systematic, transparent and replicable.
Some reviews are very different in their perspective however. A State-of-the-Art Review addresses current matters, reviewing only recent literature. It can be good for gaining a vision on what is contemporary in a field but it can also distort the view. For example, when a topic has been extensively researched in the past and may be considered ‘sated’ (e.g. compression bandaging or not fashionable e.g. elevation of the limb), then that topic may not have been recently researched and therefore not published within the recent timeframe.
Dr. Vodder School TM INTERNATIONAL Member of Dr. Vodder Academy International
Train in Canada
with certified instructors Visit our website for classes: Halifax Quebec Montreal Ottawa Toronto Waterloo London Vancouver Victoria
Lymphedema management
ONLINE Level 1 Theory Available
~ Evidence-based ~ Easy learning modules with small class sizes ~ Interactive, live classroom instruction with physicians ~ Learn precise manual skills with expert, accredited instructors ~ ISO 29990: 2010 certified training ~ CE credit available
www.vodderschool.com
info@vodderschool.com | 250-598-9862
I control my lymphedema. My lymphedema doesn’t control me. Comfortable & stylish compression garments. Fresh, never boring designs.
Another very particular type is the Critical Review. This presents, analyzes and synthesizes material from diverse sources in a conceptually new way (a new lens). It may bring together many schools of thought in a new way or aim to ‘take stock’ from a particular viewpoint. Important to note however, is that the breadth of ideas may be more important than quality of literature included e.g. a critical review of international contribution to research may focus more on the variety of countries or the range of professions involved.
Tools to critique an evidence review
Tools such as those freely available from the Critical Appraisal Skills Programme (CASP) can be very helpful (https://casp-uk.net/ casp-tools-checklists/). When reviewing any research, the first task is to identify the type of research – quantitative, qualitative or mixed research. This helps you select the appropriate template. For systematic reviews, there are ten questions under three broad headings to consider:
n Are the results of the study valid? n What are the results? n Will the results help locally?
The beauty of using a tool such as the CASP for systematic reviews is that it guides your think ing. For example, the first question asks whether the review addresses a clearly focused question. It then suggests that you consider: n The population studied n The intervention given n The outcome considered
Most questions are responded to with a yes, no or ‘can’t tell’. The latter is just as important in considering how much weight (importance) to give the systematic review. It should not be considered a scoring system however, and ultimately you will need to make a value judge ment based on your critique.
Final thoughts
All reviews start from an attempt to answer a question or solve a mystery. It is worth consid ering first whether the question itself is valid for the time and context you are in. All reviews contain some truths; consider what risks are dependent on the interpretation of those facts. There are some truths in the stories of the red dragon of the national flag of Wales and the Loch Ness Monster of Scotland yet they are both myths. In the same way, medical facts with corroborating evidence of the past have subsequently been shown to be incorrect; often because a fact has been taken out of context and then confirming bias has played its part.
The crucial thing is that the review you are reading, or considering doing yourself, is fit-for-purpose. Consider the purpose, the potential impact, the intended audience and the reality of the resources (time, skills, people used/available). As we increasingly contend with stakeholders, health service agencies and grant-holding bodies we should understand
what level of review is appropriate for our time
and money to be used efficiently. LP
A full set of references can be found at www.lymphedemapathways.ca
Advanced Lymphedema Treatment at Home
Pneumatic Compression Therapy (Lymphedema Pump) is Effective, Comfortable, and Convenient. Check your insurance! Provincial and private insurance may be available.
Learn more about Bio Compression ® Systems
1 • 800•931 •2739 paradigmmed.com
COMPRESSION & POST-SURGICAL COLLECTIONS from the Leader in “Ready To Wear” Therapeutic Torso Compression for Edema and Lymphedema
CAMISOLES SHAPEWEAR
SHORTS & CAPRIS
BRAS SHIRTS & VESTS
866.251.0076 www.wearease.com
