Science: no self-cleaning capacity

22. Science & School

text Malini Witlox illustration Bas van der Schot

Science: no self-cleaning capacity

What do Diederik Stapel, John Darsee, R.A Slutsky, Mark Spector and Jon Sudbø have in common? They were all renowned scientists, they published one article after the other, had lightning careers and their datasets were perfect. However, they were also all caught committing fraud.

“I

f you were to draw up a profile of a swindler, it would indeed meet the characteristics mentioned above. But beware because there are lots of workaholics who publish many articles, who are held in high regard, who enjoy a great career and have good datasets that aren’t fraudulent”, says Wolfgang Stroebe, Professor of Psychology at the Universities of Utrecht and Gro­ ningen.

As a result of the Stapel case, he performed research on scientific fraud. Together with the Professors Tom Postmes and Russel Spears of Groningen University he wrote an article that will soon be published in scientific magazine Perspectives on Psycholo­

'Science occasionally bumps into surprises, rare things do occur'

gical Science. The Professors looked into how often fraud occurs, whether psychology is more sensitive to fraud than other fields of science and whether peer reviews act as a deterrent when it comes to fraud.

The remarkable conclusion of their research: science has no self-cleaning capacity whatsoever. Quite the opposite of what Robert Dijkgraaf, President of the KNAW, was claiming in the weeks straight after the Stapel case. “Of course Dijkgraaf said so”, says Stroebe. “But the facts tell a different story. Swindlers are almost always unmasked by whistle-blowers or by readers of scientific articles. But, up until now there is not one known case of fraud that has been revealed by one of the two principles of self-cleaning capacity.”

The first principle Stroebe is referring to is the principle of peer reviews, whereby confreres scrutinize an author’s work.

“Sometimes peer reviewers are surprised by the findings they are presented with, but they don’t think of fraud. Fraud occurs so little that it isn’t something that is continuously in the back of your mind. However, there have been studies in which peer reviewers should have noticed that something wasn’t quite right.

For instance, in his study of genealogy, swindler John Darsee claimed that a seventeen year-old boy had four children aged eight, seven, five and four. Although this is biologically possible, it is highly unlikely that it is correct. The peer reviewers should have questioned this.”

Coincidence The second principle of self-cleaning capacity is the question of how reproducible a study is.

Among others, the studies performed by Jan Hendrik Schön and Viktor Ninov couldn’t be reproduced by other researchers.

Stroebe: "However, this doesn’t necessarily mean that a study is fraudulent. There are other reasons that play a role. It could have to do with the quality of the scientist or it could be sheer coin­ci­ dence. There is a chance that you perform research and in five out of a hundred times the outcome differs from the remaining 95 cases. Or perhaps not all the variables were clearly defined in the description of the first study, rendering it seemingly impossible to reproduce.

In the end, reproductions of studies are hardly ever published which explains why a study that can’t be reproduced isn’t made public very often. This is valuable information for the scientific community even though it doesn’t necessarily point towards fraud. A positive development, as a result of the Stapel case, is the launch of an American website (psychfiledrawer.org) where

Wolfgang Stroebe Univers 13 september 2012

Science & School .23

attempts to reproduce a study in the field of Experimental Psychology can be published.”

'As far as we know' In the discussion about fraud and swindlers one particular phrase is heard remarkably often. “As far as we know”. Because even though several big shots in the scientific community claim that fraud is an incident, nobody knows this with any certainty.

“Several studies have been conducted into fraud in the past. A meta-analysis of eighteen studies shows that two percent of the respondents admit to having tampered with data at one time or another. The most reliable study is the one conducted by the Gallup Organization. In 2008, they didn’t question American scientists on their own behavior, but they asked them whether they had encountered any fraudulent colleagues in their daily routines over the last three years. Their report concludes that 1.5 percent of all scientific research is fraudulent. This means that many cases have not been revealed yet. Usually, these aren’t cases with a magnitude such as the Stapel case but cases in which a fraudulent article was published once or twice.”

Swindlers unveiled Stroebe expects more swindlers to be unveiled over the co­­m­ing years. “First of all, peer reviewers have to realize that fraud is an option. Be alert, certainly when somebody works alone. On top of that, it’s important for datasets to be made public as this will make it less likely that a scientist will resort to fraud. Stapel, for example, gathered his own data without involving any colleagues or research trainees in the process, which is strange.”

In addition, statistical methods will play an important role in unveiling fraud. For example, the American Uri Simonsohn discovered discrepancies in Dirk Smeesters’ work by using his mathematical method. Such mathematical methods are currently being developed and we’ll be seeing more of them.

Stroebe: "But you do have to be careful. The Levelt commission is now saying that the outcomes of many of Stapel’s studies are statistically unlikely. I am concerned about the false positives of their research. We don’t know how well these methods distinguish fraudulent studies from non-fraudulent studies. If something is statistically unlikely, this doesn’t neces­sarily mean that the study is incorrect. Science occasionally bumps into such surprises, rare things do occur. I’d like to see a study whereby these methods are let loose on hundreds of studies that aren’t fraudulent (for as far as we know this with any certainty) and on hundreds of fraudulent studies.

Only then can we determine how well these methods succeed at distinguishing fraudulent studies from non-fraudulent studies.”

Headline news The two latest cases of fraud that made Dutch headline news, involved social psychologists. This has left the impression (fed by the Dutch media) that it is easier to commit fraud or that fraud is more common in the field of Psychology than it is in other scientific fields.

However, according to Stroebe this isn’t the case, on the contrary. Percentage-wise more biomedical scientists go wrong.

“In the meta-analysis performed by the Italian scientist Danielle Fanelli in 2009, it was revealed that clinical, medical and phar-

maceutical researchers deceive matters more often than any other scientists. What makes someone turn to fraud? Money can play a role. Stapel, however, was led by his desire for fame and status.”

Stapel and Stroebe aren’t strangers. They were on the same research commission of an institute where they assessed research projects and monitored the progress of trainee research pro­ jects. ”I never suspected Stapel of committing fraud. Then again, we never performed any research together. Nevertheless, I was shocked. You often hear people say that they can remember where they were at the time when they heard that Kennedy was shot. Well, I can remember exactly where I was when I heard that Diederik had committed fraud. I was informed by an e-mail from a colleague. Incredible.”

For a detailed overview of fraudulent scientists (in Dutch) visit our website: http://universonline.nl/2012/06/27/fraudeurs-op-een-rijtje/

Univers 13 september 2012