Lab+Life Scientist Jun 2015

Page 40

selective reporting Lauren Davis

Are you tweaking your experiments? Australian researchers have stated that some scientists are unknowingly tweaking experiments and analysis methods in order to increase their chances of obtaining easily publishable results. Their study has been printed in the journal PLOS Biology, no doubt making some readers wonder if it too has been altered for publication!


he study examined a type of bias called

p-hacking, which occurs when “researchers try out

that p-hacking is happening throughout the life sciences”.

“They might look at their results before an experiment is finished or explore their data with

several statistical analyses and/or data eligibility

Dr Head suggested that “pressure to publish”

specifications and then selectively report those

may be driving this bias, noting along with her

“Many researchers are not aware that certain

that produce significant results”, according to the

co-authors that “there is good evidence that

methods could make some results seem more

authors. While such actions may be conscious or

journals, especially prestigious ones with higher

important than they are. They are just genuinely

unconscious on the part of the researcher, the end

impact factors, disproportionately publish

excited about finding something new and

result is the same - data is analysed multiple times

statistically significant results”. There is thus an


or in multiple ways until a desired result is reached.

incentive for researchers to selectively pursue and

The authors acknowledge that p-hacking is

lots of different statistical methods.

The study used text mining to extract p-values

attempt to publish such results, with the study

a serious issue, stating that the “publication of

- a number that indicates how likely it is that a

finding a high number of p-values that were

false positives hinders scientific progress”. Many

result occurs by chance - from more than 100,000

only just over the traditional threshold that most

scientists may be uninterested in replicating

research papers in the PubMed database, spanning

scientists call statistically significant.

previous (supposed unbiased) studies, while others

many scientific disciplines, including medicine,

“This suggests that some scientists adjust their

biology and psychology. According to lead author

experimental design, datasets or statistical methods

Dr Megan Head, from the ANU Research School

until they get a result that crosses the significance

of Biology, the researchers “found evidence

threshold,” Dr Head said.

may pursue fruitless research programs based entirely off their results. Even when scientists review evidence by combining the results from multiple studies - a method called meta-analysis - this procedure will be compromised if the studies being synthesised “do not reflect the true distribution of effect sizes”, according to the authors. They do concede, however, that p-hacking “probably does not drastically alter scientific consensuses drawn from meta-analyses”. The authors have made a series of recommendations to prevent p-hacking from occurring. They suggest researchers adhere to common analysis standards (performed blind wherever possible) and place greater emphasis on the quality of research methods rather than the significance of the findings. Journals, meanwhile, are encouraged to provide clear and detailed

Dr Megan Head in her evolutionary biology lab at the ANU Research School of Biology. Image credit: Regina Vega-Trejo.

40 | LAB+LIFE SCIENTIST - June 2015

guidelines for the full reporting of data analyses and results. |