EPP sloppy science 24 jan 2014

Actually this talk is on power mechanisms: what counts as good science? – and who decides? And: who counts as a good scientist? – and who decides?

The H-index (see http://en.wikipedia.org/wiki/H-index) is widely used as an indicator of scientific quality, but as such it is also contested, see for instance: http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individualresearchers-the-debate-in-berlin/ and http://citationcultures.weblog.leidenuniv.nl/2011/09/26/still-using-the-hirsch-indexdon-t/ In this talk the H-index stands for ‘prestige and pressure by using output criteria’

1

This is a signal of confirmation bias, the production of good results. In this case, the confirmation bias is produced at the level of design (see for examples of other types of confirmation bias: Lesley John et al. 2012, Measuring the prevalence of QRP’s …

In his lecture in Oxford, the 32 year old Stapel opened up about his research practices – and no one debated them at the time (although they were definitely questionable research practices – QRP’s). A few months later, his talk was published in the European Bulletin of Social Psychology (2000, 14, 4-27) as ‘Moving from fads and fashions to integration’. The Levelt Committee that investigated his fraud case, used the quote to illustrate the deviant norms within this branch of experimental social psychology. See also my discussion of this example in De publicatiefabriek, p. 57-63.

This raises the question: is there only one scientific method or are there many?

Obviously, the latter is the case, but once someone claims to stick to the experimental method and rigorous hypothesis testing, deviations might rightfully be considered to be QRP’s.

Science is automatically interpreted as natural science; the Dutch word ‘wetenschap’ automatically includes the arts or humanities. Studying inanimate objects such as quarks or celestial bodies, requires different methods than studying mental functions, social processes, historical events or cultural differences. Even if basic laws in human or social sciences could be established, the complexity of actual human life would require a continuous checking and balancing of initial conditions and ‘confounding’ variables. Giddens doubts if such laws can be found at all, because human beings are intelligent actors that would confound these laws all the time and have their own interpretations of what’s going on (see quote on next slide).

3

In human sciences we should not just be trying to find basic mechanisms, but also study the context (including the interpretations our research subjects give of this context – even if (or maybe: especially if) this context is an experimental design. Ideally, to come up with results that have ‘street credibility’ (which is especially useful in all sorts of interventions) we need to combine all types of empirical evidence.

Giddens states that human beings are ‘natural’ social scientists (otherwise human society would have ceased to exist long ago). Although their fact checking isn’t as rigorous as that of ‘real’ social scientists, the latter need to come up with good results (and interpretations!) in order to convince their equally intelligent (but less trained) fellow human beings.

4

Although officials in the academic world tend to formally agree with Van Oostrom, in daily academic life long publication lists still manage to impress outsiders – and even insiders, as the recent case of regional economist Peter Nijkamp illustrates (‘there is nothing wrong with my publication performance’, he wrote). Apart from the truism that productivity is not the same as quality, the ‘publication explosion’ might damage science as a whole, because it becomes harder to integrate knowledge or, as social psychologist Naomi Ellemers put it’ to ‘connect the dots’ (see her article in the European Journal of Social Psychology 43 (1), 1-8, or the reworked version of it in De Psycholoog (dec. 2013).

5

Most people go into science to do substantial work rather than make a career by ‘organizing publications and grants’. Nowadays, the latter seems to be necessary to achieve the former. Keep in mind, though, that this is not what doing science really is about.

Think for yourself, don’t let others do that for you. To acquire the necessary personal leverage, it is useful to familiarize yourself with the history of your discipline and with philosophy of science. For the field of the human / social sciences you might consult my book Over de grenzen van disciplines, in which I discuss the current situation of the human science disciplines from a historical and philosophical perspective.

6

Over de grenzen van disciplines discusses the position of the human sciences between natural sciences and humanities, and in their complex position between theory and practice. Various new perspectives are presented, such as the contributions by Bent Flyvbjerg, Keith Sawyer en Peter Manicas. The book ends with a chapter on future perspectives of the human sciences, including the ‘industrialization’ of science, the equation of productivity and quality and the ever growing ‘governance’ of academic work. De publicatiefabriek can be seen as a sequel, which uses the Stapel fraud case to give an in depth analysis of problems in current academic work. The book has its own weblog: http://depublicatiefabriek.blogspot.nl/