The Pursuit of Objectivity in Psychology

M AT T I A S D E S M E T

THE PURSUIT OF

OBJECTIVITY I N PSYCHOLOGY

–5–

CONTENT

7 Foreword 15 Introduction

23

CHAPTER 1

The measurement problem from a conceptual perspective

CHAPTER 2

45

The measurement problem from an empirical perspective

CHAPTER 3

61

Examples

CHAPTER 4

77

Reorientations

91 Conclusion 99 References

–7–

FOREWORD A reader deserves to know the circumstances in which a book was written. In 2003, after completing my training as a psychologist, I was given the opportunity to do a PhD at the Department of Psychoanalysis at Ghent University, Belgium. The research project proposed to me was quite mainstream: I would measure personality styles and symptomatology in a sample of depressed patients and statistically test hypotheses about associations between these variables. While gathering the metric data, however, I also did some interviewing with the patients. More often than not, it struck me that the outcome of the measurement procedures did not match my impressions during the interviews. This phenomenon immediately sparked my interest. With my promoter’s permission I decided to shift the focus of my PhD from the testing of hypotheses to investigating the validity of the measuring instruments. And so it came to pass that, having arrived at the department with a passion for Lacanian psychoanalysis, I started following an additional two-year master in statistics and psychometrics, and immersed myself in the methodological background of nomothetic psychology. By the time I had completed my PhD, about four years later, I was more than ever convinced of one thing: there is a measurement problem in psychology, and it is profound.

Later on, I reoriented my research in a more clinical direction, but from time to time I still tried to grasp the essence of the measurement problem in psychology. This book represents my current thinking on this issue. Throughout the writing, it became clear to me that it is not sufficient to tinker with psychological methodology a little bit here and there. Instead, we need a radical shift Fo r e wo r d

– 8 –

towards the single case as the primary unit of analysis, a reappraisal of narrative description, an ambition to describe psychic phenomena in a mathematical way, and a measurement paradigm that is based on quantification of naturalistic speech rather than on the registration of specific ‘responses’ to specific ‘test stimuli’.

To be honest: my skepticism about classical psychological methodology did not start during my PhD. During my initial training as a psychologist, I already had the intuition that mainstream psychological methods were ill-suited to the phenomenon of subjectivity (i.e. the core object of psychological research). It seemed to me that the field of psychology had made a monumental mistake by neglecting the single case and relying exclusively on metric description. This book offers a logical and empirical elaboration of this initial intuition and I confess humbly to the reader, before he starts reading, that in several respects its conclusions confirm my a priori subjective preferences. Nevertheless, I proceeded with as much reason and honesty as I am capable of and I hope that the reader will judge the results in the same spirit.

Overall, this raises the pertinent question as to how ‘objective’ one can be as a researcher. In the introduction to this book, I will refer to the current crisis in the academic world, more specifically to complications relating to the replication of research findings and reports of sloppiness and fraud. It seems that objectivity in science has been dramatically overestimated. One might call this a perversion of the true spirit of science. Yet even if a researcher is free of any such inclination, he often proves to be far less rational and objective than he might imagine. This is not only due to the intrinsic lack of rationality in man himself; it is also a characteristic of the objects he investigates. If science in the twentieth century has shown us anything, it is that God does indeed play with dice. The Age of Reason – the tradition of the

–9–

Enlightenment – ultimately shows us in an unequivocal way that the world and the things in it are not rational (also see Desmet, 2019). Or to put it in the words of Heisenberg (1974): “Not only is the world stranger than we think; it is stranger than we can think.”

That is why Niels Bohr (1933, in Pranger, 1972) contended that if science wants to grasp the real, scholars must learn to think illogically and use language as poetry.

At the same time, the last centuries have shown us that reason is nevertheless effective. It leaves its traces, for instance, in the world that is subject to man’s reasoning. Nobody can ignore that. After all, man went to the moon. (After all, the oceans are full of garbage, another might say). And no matter how irrational we might be, reason also leaves its traces in us, in the reasoning subjects themselves. For myself, I can say that throughout the process of writing this book, regardless of the fact that its conclusions confirmed some of my subjective preferences, I have applied reason to the best of my ability. It was this same reason that also forced me to declare some of those preferences to be incorrect.

I hope that reading this book and following its line of reasoning will have a similar effect on the reader and that it will not leave him in exactly the same way as it found him.

While this book does not contain many pages, it has been a long time in the making. I rewrote the text on numerous occasions, making it longer and then shortening it again, making it more complex and then reducing the complexity again. I presented fragments of it at several conferences, sometimes for audiences as small as two or three people, sometimes as large as a few hundred. I usually had the impression that it was received with enthusiasm, that it revealed something to the audience that they had been sensing while doing their own research but had never been able to grasp fully. In any case, the responses were Fo r e wo r d

– 10 –

sufficiently encouraging for me to continue with my writing. I hasten to add that the ideas presented in this book also invoked adverse reactions. On rare occasions, scholars who identify with the mainstream discourse in academic psychology reacted quite critically. I believe that this might have to do with the straightforward manner in which this book addresses some of the dominant paradigm’s methodological shortcomings.

“An evening everyone agrees, is a lost evening”, Albert Einstein said. Science is articulated through conflict and discussion. The courage to formulate your own opinion in a non-conformist way, to the best of your own understanding, without minding too much whether the other-opponent might like it or not, but being fully aware that you will need this other who disagrees with you to transcend yourself and to reach a higher level of understanding.

These are the two necessary preconditions for true scientific progress. It is my hope that the proponents of the dominant paradigm in psychology will consider this book as an invitation to engage in a dialectical exchange, which will eventually lead, not to the victory of one of the opponents and the defeat of the other, but to the emergence of new ideas which transcend oppositions. In a few years, we might consider writing a new book together in which we present the results of our exchange and as such contribute to a solution for the crisis psychology finds itself in today.

The Department of Psychoanalysis at Ghent University offered a fertile academic context for the writing of this book. I wish to express my gratitude to my colleagues – in particular to Paul Verhaeghe and Stijn Vanheule, the former and the current head of the department – for their willingness to set a psychoanalytic course in a time when such is not the easiest choice. Back in the summer of 2003, when I had just been taken on, the department had little or no tradition in mainstream empirical research. As was

– 11 –

often the case in psychoanalytic culture, its academic activities were largely confined to conceptual studies and classical clinical case studies. However, there was a general feeling among psychoanalysts working in the academic context that without a more empirical approach psychoanalysis would be doomed to perish at university. Hence the department faced the task of establishing an empirical research program without losing its non-conformist, psychoanalytic nature. In those early days it was not always easy to disentangle the ‘strategic’ and ‘authentic’ motives for engaging in empirical research.

My personal position towards empirical research deserves some further clarification. Beyond mere tactical considerations, I have always believed in the potential of empirical research to make a quintessential and irreplaceable contribution to psychoanalysis.

For me, the historical project of psychoanalysis is inextricably rooted in the tradition of the Enlightenment. Its place in this tradition is exceptional, since it is not possible for psychoanalysis as a discipline to be reduced to its purely scientific dimension (at the expense, for instance, of its ethical dimension). Even so, loyalty to reason and respect for empirical (clinical) observation – the two cornerstones of the ideology of the Enlightenment – are essential to psychoanalysis, as was perceived by Freud right from the very beginning.

At a more personal level, and irrespective of the compatibility or otherwise of psychoanalysis with an empirical approach, I also have a feeling for empirical science. Not so much because of its practical applications or its supposed capacity to improve the human condition by sophisticated manipulation of man’s material context. On the contrary, I am rather skeptical in this respect. I am attracted to science for its revelatory capacity, for the perspectives it offers in the quest for knowledge an sich and, Fo r e wo r d

– 12 –

beyond that, for truth. In my interpretation, the empirical branch of the Enlightenment is grounded in a sublime ethical maxim, which boils down to this: let things speak for themselves, listen to what they have to say without silencing them by prejudice and pre-established theory. Science is born from the courage not to know, the courage not to understand, from the capacity to tolerate uncertainty. This position of not knowing is the only one which eventually allows us to hope for the emergence of a new knowing.

From this perspective, psychoanalysis and the tradition of the Enlightenment are perfectly in line with each other. What else is psychoanalysis other than listening to a specific set of empirical facts – i.e. a subject’s speech – as open-mindedly as possible?

While writing this book, I presented my ideas time and time again to others. Reitske Meganck read numerous preliminary versions of the manuscript, with the rare combination of modesty and sharp-wittedness that is so typical of her. It was only when she nodded her head in approval that I dared to consider a certain part of the text as finished. Together with Jochem Willemsen and Ruth Inslegers, she tolerated my innumerable tales of measurement error, not only at the department, but also at conferences, in airports, at dinner parties and during long walks in nature. This book will forever remind me of the foreign countries the four of us visited together. The final draft was written while preparing two lectures for the University of Copenhagen, the first during the summer of 2016, the second a year later. It was Katrine Zeuthen who invited me to give these lectures. Her interest, warm enthusiasm, and intuitive understanding of the point I tried to make, enabled me to finalize the text I had been developing for such a long time. I must also extend my gratitude to other colleagues who took the time to read preliminary versions of the text, including Jan Celie, Maarten De Schryver, Jasper Feyaerts, Juri Krivzov,

– 13 –

Clare Murphy, Yves Rosseel, David Schrans, Femke Truijens and Dana Tzur-Bittan. Some of their comments led to adaptations and adjustments, others did not. But the mere fact that they were reading what I wrote was sufficient for me to carry on writing and bring this work to its conclusion.

Mattias Desmet, summer 2018, Meigem.

Fo r e wo r d

– 15 –

INTRODUCTION For about a decade, the academic world has been gripped by a feeling of crisis, which set in following reports of generalized replication failure, widespread methodological errors, and several cases of outright fraud (e.g. Allison, Brown, George, & Kaiser, 2016; Baker, 2016). It was particularly the latter category that caught the eye. Scientific imagery was manipulated (e.g. Bik, Casadevall, & Fang, 2016), archaeological objects counterfeited (e.g. Jarus, 2018) and the cloning of embryos fabricated (e.g. Souza & Caitite, 2009). Some claimed successful skin-transplantation in mice yet were discovered imitating the intervention by painting the skins of their test animals (Hixson, 1976, cited in Eisner, 2018). Others even pretended to have found the evolutionary missing link on the basis of a self-made skull using pieces of ape and human bone (Degroote, Flink, Abbas, et al., 2016).

It is possible to find examples of this kind of deception at all times and in all places, but some review studies describe a sharp increase of questionable research practices during the last decade (e.g. Bik, Casadevall, & Fang, 2016). In particular, certain milder forms of scientific misconduct seem to have reached truly epidemic proportions. One systematic review (Fanelli, 2009) yielded conservative estimates as high as 72% for the proportion of researchers who would be prepared in one way or another to engage in strategies that distort scientific observations.

Less spectacular but even more problematic were a series of studies reporting that research findings in several academic disciplines suffered from a generalized lack of replicability and reproducibility (i.e. repeatability of observations in the same dataset).

In cancer research, for example, observed replication failure rates Introduction

– 16 –

were about 60% (Begley & Ellis, 2012), whereas in economics the figure was about 50% (Chang & Li, 2015) and in biomedical science even as high as 85% (Begley & Ioannidis, 2014). Taken together, the extent of the reported problems was such that it has led some to conclude that across scientific disciplines “most published research findings are false” (Ioannidis, 2005).

It is slightly ironic – yet highly significant – that the figures on replication rates also vary substantially across publications. In other words: the findings on limited replicability lack replicability themselves. Perhaps this single observation, more than any other, allows us to gauge the depth of the crisis.

In some respects, the field of psychology seems to be particularly hard hit. Several cases of far-reaching fraud, sometimes to the extent that entire datasets turned out to be fabricated, have surfaced (e.g. Vogel, 2011). Furthermore, during the last two decades more and more scholars, including Noble Prize winner Daniel Kahneman (2012), alarmed the research community by contending that psychological studies suffer from a bewildering methodological sloppiness (e.g. Enserink, 2012; Simonsohn, 2012; Toomela, 2007; Schwartz, 2009; Vogel, 2011; Wagoner, 2007) and from a generalized lack of replicability and reproducibility (e.g. Bartlett, 2014; Bohannon, 2014; Ritchie, Wiseman, & French, 2012; Roediger, 2012; Stevens, 2017). These claims were confirmed by large-scale research that showed how attempts to replicate failed in the majority of cases (Open Science Collaboration, 2015). The inevitable conclusion therefore seems to be that in psychology there are “widespread and systematic problems in the way researchers produce, evaluate, and report knowledge” (Rouder, 2016, p. 1062).

– 17 –

As some authors have highlighted (Maniadis & Tufano, 2017, p. 200), the replicability of findings has not been investigated as thoroughly and as systematically in other academic disciplines as in psychology. This implies that there are no firm grounds to conclude that the problems in psychology are necessarily more severe than those in a wide variety of other academic disciplines.

To a certain extent, the public dismay that was targeted at psychology might therefore have more to do with ungrounded prejudice than with sound, critical argumentation (e.g. Lilienfeld, 2017).

No matter how true this might or might not be, it does not relieve psychology – like any other scientific discipline – of its responsibility to tackle these emerging problems and develop concrete plans to manage them (Lilienfeld, 2012). In a certain sense, psychology has come up with such plans. The crisis gave rise to the so-called ‘evidentiary value movement’, representing a more or less coordinated set of initiatives designed to counter the crisis (Nosek, Spies, & Motiel, 2012; Asendorpf, Conner, De Fruyt, et al., 2013). A series of putative causes of sloppiness and lack of replicability were described, which included researchers’ carelessness, journals’ unwillingness to publish replication studies and null-findings (Levenson, 2017, p. 675), researchers’ dependency on grants (Lilienfeld, 2017, p. 661), and the ‘publish or perish’ culture in the academic world (Lilienfeld, 2017, p. 661).

The remedies primarily proposed a focus on investing more in replication studies and promoting a more open approach to science, in particular through the pre-registration of hypotheses and research questions and by making data available online (see, for instance, agendaforopenresearch.org, Morey et al., 2014).

It is noteworthy that the crisis is almost always attributed to a lack of strictness in the application of research methods. In general, academic psychology seems keen to console itself with the Introduction

– 18 –

idea that a stricter and more conscientious adherence to research methods will be enough to turn the crisis into a positive experience, which will ultimately have a strengthening effect on its scientific status (e.g. Fiske, 2017; Lilienfeld, 2017).

These attempts to promote a more conscientious application of research methods can only be applauded. At the same time, however, they might not be sufficient to solve the problem altogether. There is not only a problem with the application of the methods, but also, more fundamentally, with the methods themselves. At least one important contributory factor appears to remain unrecognized: the problematic reliability and validity of psychological measures. At best, papers analyzing the replicability issue only mention this factor in the margin (e.g. Asendorpf et al., 2013, p. 111). In a certain sense, this is strange. In the decades preceding the onset of the crisis, and long before the advent of the public pressure resulting from it, several scholars warned the research community that the validity and the reliability of the measures being used were highly problematic and that the effect on research practice was disastrous (e.g. Borsboom, 2006, p. 426; De Shon, 1998; Paulhus, 1998; Essex and Smythe, 1999; and Michell, 2000). Their skeptical evaluation contrasted sharply with the countless other papers that in their method and measures sections described the validity of the measures used in psychology as ‘acceptable’, ‘good’, or even ‘excellent’. Consequently, the warnings went largely unheeded. In this book, I will join the side of the skeptics and argue that the contemporary crisis in psychology in many respects represents the return of an all-too-frequently denied problem at the level of measurement methods.

This neglect of measurement issues in the analysis of psychology’s crisis is quite remarkable. In a certain sense, measurement problems underlie all the different problems in psychological

– 19 –

research outlined above. Lack of validity and reliability of measurements implies lack of replicability and ambiguous observations; ambiguous observations, in turn, open up a space in which methodological strictness is easily replaced by sloppiness; and sloppiness, finally, entails the blurring of ethical boundaries (e.g. Boutron, Dutton, Ravaud, & Altman, 2010; De Los Reyes, Kundey & Wang, 2011, p. 831). In this book, I will argue that the measurement problem compromises the scientific status of psychology and it is no exaggeration to say that academic psychology’s future depends upon the success or failure of attempts to deal with this problem.

Finding a solution to this methodological deadlock becomes even more urgent if one considers the scientific importance of psychology. From many perspectives, the coming era might well be the era of psychology. And this is not just a psychologist’s wishful thinking. Scientists from other disciplines – anthropologists, philosophers, ethnologists, biologists, physicists, etc. – have often described the profound impact of subjective experience on phenomena that were previously conceived in purely materialist-mechanist terms, including perception (Merleau-Ponty, 1945), reflexes, pain, and motoric behavior (e.g. Mauss, 1934), physiological and anatomical characteristics (e.g. Chauvin, 1941; Harrison-Matthews, 1939), and even the location of material particles in space and time (e.g. Heisenberg, 1927). Furthermore, the omnipresence of placebo effects in medical interventions (Wampold, Minami, Tierney, et al., 2005) and the efficacy of hypnotic anesthesia in surgery (e.g. Faymonville, Meurisse, & Fissette, 1999) show in an unambiguous way that psychology is not only intellectually fascinating, but also has a tremendous practical and clinical relevance, of which we are only starting to realize the potential.

Introduction

– 20 –

Psychology therefore needs to accept its historical responsibility and find a solution to the methodological impasse in which it now finds itself. In this book, I will review some of the literature and argue that, in order to effectively solve the current crisis, psychology needs to reconsider the way in which it has (mis)attuned its methodology, in particular its measurement methods, to the objects it investigates. In doing so, I will base my case on science’s two main pillars, taking both a logical-conceptual and an empirical point of view. Chapter I will situate measurement among other types of scientific description and present some logical-conceptual reflections on the nature of psychological objects, showing that metric description (measurement) only applies to a subset of these objects. Chapter II will confront these logical-empirical reflections with observations in validity and reliability research, and argue that, contrary to what the majority of researchers claim in their empirical papers, such research actually confirms the problematic status of mainstream psychological measurement procedures. Chapter III will present some concrete examples of research findings, which both illustrate and challenge the line of reasoning presented in the previous chapters. Chapter IV, finally, will consider alternative methods of measurement based on the quantification of language and will explain how these methods might contribute to a more profound solution of psychology’s current crisis.

In my analysis of the measurement problem, I will be straightforward, frank and remorseless. This might be inconvenient for some, yet in my opinion it is necessary, and I appeal to my readers to search earnestly, together with me, to find the origins and solutions of the problem we need to solve. Since this book not only addresses psychometricians but is also intended to contribute to the broadest possible change in attitude within the research

– 21 –

community, I will strive to present this complex matter as simply and as accessibly as I can. Furthermore, since I believe that overcoming the present crisis might also represent a moment of Aufhebung or sublation for psychology, I will not eschew from reviewing literature that deals with many different, often conflicting, theoretical and methodological backgrounds. And finally, since arguments do not necessarily become any less valid with age, I will refer to literature from both the early days of psychology and from more recent times.

Introduction

C H A P TER I

The measurement problem from a conceptual perspective

– 25 –

Let me start these reflections on the measurement problem by drawing attention to the fact that measurement is a type of description. Description boils down to the translation of the ‘raw’ phenomena or ‘objects’ of investigation into research data, and this can be done in different ways.

First, there is narrative description, in which one describes the object of investigation with words. While at first sight such description seems to lack the level of exactness and sophistication required in science, it is nonetheless clear that it has played a major role in virtually every major scientific achievement, whether it be the work of Copernicus (1543), Linnaeus (1753), Darwin (1859) or any other groundbreaking scholar.

Second, there is description by means of (visual) images.

Historical examples of this type of description include the detailed drawings of human anatomy by Vesalius (1543), the drawings and photographs of animals in the work of Tinbergen (e.g.

1948), and the x-ray crystallography pictures (Wilkins & Randall, 1953) on the basis of which Watson and Crick (1953) discovered the double helix of DNA. Third, there is measurement or quantitative description.

Measuring an object is essentially a matter of counting measurement units, or, in other words, situating this object alongside a unidimensional scale consisting of a measurement unit multiplied by a real number ( Joint Committee for Guides in Metrology, 2012). This implies, strictly speaking, that only the unidimensional characteristics of objects can be measured. For instance, one cannot measure a car an sich. One first has to determine its THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 26 –

unidimensional characteristics, such as length or weight, then choose a stable and invariant measurement unit, and finally count the number of units in the object characteristic to be measured.

While common sense would seem to suggest that (sophisticated) measurement should offer a quasi-absolute assessment of an object, it is important to note that in reality it is always relative to the scale unit used. Mandelbrot (1967) illustrated this in a very tangible way in a paper entitled ‘How long is the coast of Britain?’ If one looks at the British coastline, one does not see a straight line, nor even a smooth, curved line, but rather a twisting, meandering line. Measured with a scale unit of 200 km, the length of this line is 2400 kilometers; with a scale unit of 50 km, the length become 3400 kilometers (Figure 1). Mandelbrot was able to demonstrate that “as the scale becomes smaller, the measured length of the coastline rises without limit, bays and peninsulas revealing ever-smaller sub-bays and sub-peninsulas” (Gleick, 1987, p. 96). Richardson (1961) underlined the manner in which this can lead to practical problems. He checked encyclopedias in Spain, Portugal, the Netherlands and Belgium and found that the reported length of the common borders showed discrepancies of about 20%, due to different scales used.

– 27 –

F I G URE 1

The measured length of the coast of Britain is 2400 km (approx.) with a scale unit of 200 km (left side of figure) and 3400 km (approx.) with a scale unit of 50 km (right side of figure) (image retrieved from Van de Sande, 2004).

Fourth, there is mathematical description (e.g. Prusinkiewicz & Fowler, 1995, p. 163), which is often overlooked or confused with quantitative description and which consists of determining the formula of a phenomenon. The Fibonacci formula, for instance, determines a spiral-shaped curve, which shows the proportions of the golden ratio and exactly matches a host of curves in nature, such as those found in sunflowers, seashells, tornados and galaxies. We could give numerous other examples; for instance, of fractal formulas that determine a variety of highly complex natural forms, such as seashells (Meinhardt, 1995), seaweeds, sponges and corals (Kaandorp & Kübler, 2001), and a variety of plants (Prusinkiewicz & Lindemayer, 1990). While the first three types of description are ultimately relative and imprecise – narrative description is possible using many different combinations of THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 28 –

words, photographs or pictures can be made from many different perspectives, and measurement is always dependent on the scale used – mathematical description alone seems to touch the absolute. There is just a single mathematical formula to determine a mathematical shape. This gives mathematical description a special aura and status, making it reminiscent of Plato’s universal forms, as if it is the finishing touch, the crowning glory of the scientific process.

Nevertheless, it is clear that all four types of description are necessary for effective science. Mathematical description would be a meaningless abstraction without the other descriptors. The ultimate objective of science must therefore be to ‘knot’ together the different types of description in constant reciprocal reference.

The more solidly they are knotted together, the more consummate the scientific theory. In many respects, such a conception is preferable to a conception in terms of explanation. Explanation is always causal explanation. However, the ultimate cause of things regresses without limit. That is why science, in the final analysis, fails to explain. Something similar was voiced by von Neumann (1955): “The sciences do not try to explain; they hardly even try to interpret. They mainly make models. By a model is meant a mathematical construct, which, with the addition of certain verbal interpretations, describes observed phenomena.”

Deciding which type of description is appropriate depends on the characteristics of the objects to be described. Consequently, we should ask ourselves: what are the objects of psychology and what characteristics do they possess? We can distinguish between numerous psychic phenomena, such as emotions, cognitions, memory, coping mechanisms, psychological disorders, personality traits, etc. That being said, there is a certain tendency in psychology to refrain from studying psychic phenomena and to

– 29 –

focus instead on external correlates at the biochemical, behavioral or neural level: “[ever since] … the psyche arrived in the role of the object of investigation, everything has been done … to deny its reality and its status as the appropriate subject matter of psychology.

Psychology is a science that has, quite successfully, denied its own existence. It has tried to get rid of itself by offering itself to physiology, sociology, computer science, and currently to neuroscience. Psychology is a science that is afraid of itself as a science!” (Valsinere, 2012, p. xiii, cited in Schwarz, 2014, p. 214). It is clear that this has the potential to lead us astray, if only because, when we want to know whether a certain biochemical or neural process is indicative of a certain psychic phenomenon, we first and foremost need to define and describe the phenomenon itself. Therefore, in the broadest sense, the core objects of psychology are always what we might call the psychic apparatus and psychic experience.

Many perspectives can be taken on the psychic apparatus, yet for the purpose of this book I will refer to a network conception. Virtually all major psychological theories – from the early conceptions of Mesmer, Fechner, Janet, Charcot and Freud (see Ellenberger, 1970, pp. 145-149) to more recent cognitivist theorizations (e.g. Collins & Loftus, 1975) – maintain such a conception at one level or another. The core structure of the network is always an associated set of phonological-linguistic nodes (e.g. Stella, Beckage, & Brede, 2017) which, in turn, is associated with perceptual images (e.g. Bernstein, Clarke-Stewart, Roy, et al., 1994, p. 289) and bodily experiences (e.g. Galetzka, 2017; Bernstein et al., 1994, p. 308). While certain network aspects are the same across smaller or larger social groups – for instance, across everybody speaking the same language – others are based on a subject’s historical and idiosyncratic set of experiences and are therefore subject-specific.

THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 30 –

The network apparatus can be activated by triggers as varied as stimuli arising from memory, sensory images from the outer world, or the speech of other subjects. Whatever its origin, as soon as a trigger activates a node, the activation spreads in complex ways throughout the network and activates chains of phonological nodes, which manifest as internal thoughts that might or might not be externalized as speech. And as these chains of phonological entities manifest in subjective experience, so layers of meaning arise underneath. As early as 1890, one of the founding fathers of psychology, William James, remarked that meaning typically arises “in pulses, after clauses or sentences are finished” ( James, 1890a, p. 182) and that the meaning of a phonological unit, of which the most prototypical example is a word, “… consists of other words aroused, forming the so-called definition” ( James, 1890a, p. 160).

For instance, the node ‘hit’ has a different meaning in the chain of words ‘She hit him’ in comparison with ‘Her song became a hit’ or ‘Hit the road’.

This all implies that the emergence of meaning is highly volatile and unpredictable. Minor differences at the level of incoming stimuli can provoke radical shifts at the level of meaning and psychic experience. Throughout this book, I will also refer to the distinction between the phonological nodes in networks and the meaning arising from the network by means of the more appropriate conceptual distinction between signifier and signification, as expounded in the linguistic theory of Ferdinand De Saussure (1906, p. 75). The term ‘signifier’ (i.e. a unit that signifies something) refers to the set of phonological entities or acoustic images that form the material basis of language (see De Saussure, 1906, p.

75), whereas the term ‘signification’ refers to the set of signifiers that defines another signifier.

– 31 –

As activation spreads throughout the network and meaning emerges, network nodes consisting of perceptual images (e.g. visual images, James, 1890a, pp. 360-361) and bodily experiences (e.g. sensations or vital impulses, James, 1890b, pp. 301-315) are also activated, so that rich, embodied, subjective experience occurs.

The network actually delivers the total set of units a subject can use to attribute meaning (also see Bernstein et al., 1994, p. 307) and to generate psychic experiences. As such, it represents not only the basic structures of memory, but is also deemed to determine the set of psychic attributes that constitute what is usually called the personality of a subject, including emotional and affective responses, cognitive styles, motivation, social behavior, coping strategies and typical ways of appraising stimuli (Wells & Matthews, 1994), as well as more vital response tendencies such as aggression (Bushman, 1996), psychological symptoms, complaints and disorders (Bringmann, Vissers, Wichers, et al., 2013). Nearly all objects of psychological research can thus be situated within a network discourse, which justifies using it as a central reference point throughout this book.

This description of the psychic apparatus and the way in which psychic experience and meaning arise in it, shows us that the psychic system, in all respects, is a complex and dynamic system. This is nothing new; many others have come to the same conclusion (for example, Guastello, Koopmans, & Pincus, 2009; Salvatore, 2016, pp. 236-240). It is interesting to consider the consequences of this for descriptive methodology, starting from the dichotomy between mechanical systems on the one hand and complex dynamical systems on the other hand. A mechanical system interacts with its environment in a linear, invariant and predictable way according to a set of mechanical laws. The behavior of a ball that drops to the ground, for instance, is determined by mechanical THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 32 –

laws that predict how it will accelerate, hit the ground, how high and in what direction it will bounce, and so on. In this respect, mechanical systems – from the simplest to the most complicated – are characterized by deterministic predictability.

In contrast, complex dynamical systems possess the most fascinating characteristic of deterministic unpredictability. A wellknown example is convection rolls, which emerge when liquid or gas is heated. As the substance is heated, the rolls first behave in a stable, repetitive way. But when the temperature increases and reaches a critical limit, the rolls start to behave in a chaotic way. Interestingly, the behavior of the rolls in this phase can be described by a set of three non-linear equations with three unknowns and hence is strictly deterministic (Lorenz, 1963).

At the same time, however, these mathematical equations never reach a final, stable solution and, consequently, the outcome is also non-periodic and fundamentally unpredictable. This unpredictability can also be understood from a more pragmatic perspective, based on the property of complex dynamical systems to display sensitivity to initial conditions, which means that the impact of the smallest factors can lead to radical differences in the way the entire system manifests itself. Since decisive differences can be infinitely small, measurement of the conditions can never be accurate enough to predict how the system will behave (Gleick, 1987, p. 19). A difference in the behavior of one liquid molecule at the beginning of the heating, for instance, can lead to the manifestation of radically different convection rolls. It is clear that the characteristics of deterministic unpredictability also apply to psychic experience and to the phenomenon of meaning. Psychic experience is strictly determined, in the sense that meaning does not arise in an arbitrary way. But because even the smallest details of a situation can lead to radical shifts in the meaning given to it by

– 33 –

a subject – the blink of an eye, a faint smile or an almost inaudi­ble cough can radically change the meaning one attributes to another person’s entire discourse – it can never be fully predicted what meaning will eventually emerge (Salvatore, 2016, p. 239).

The example of convection rolls is also illustrative of two other characteristics of complex dynamical systems that are particularly relevant for our line of reasoning. First, complex systems tend to coalesce with their environment (e.g. Bishop, 2011, p. 115), as if system and environment are both contained within an overarching supra-system. It is hard to determine, for instance, whether the behavior of a convection roll is due to the dynamics of the roll itself or to the currents in the liquid or gas in which it manifests.

At the same time, the roll organizes and is organized by its environment. In a quasi-identical way, it is hard to determine whether a subject’s behavior is due to the subject itself or to its social-psychological environment. In empirical research, this is illustrated by so-called researcher-demand or experimenter-expectancy effects, referring to the impact of the researcher’s expectations and intentions on participants’ responses. This has been described in papers with illustrative titles, such as ‘Telling what they want to know’ (Norenzayan & Schwarz, 1999) and ‘Self-reports: how the questions shape the answer’ (Schwarz, 1999). Danziger (1990, p. 8) rightly remarks that, although psychology now recognizes this problem and has even coined terms to describe it, nothing has really happened to change its methodology and develop designs in which subjects can be truly studied in their co-existence with other subjects. Moreover, it is remarkable that the so-called major sources of ‘bias’ in psychological measurement – such as acquiescence, compliance, social desirability and impression management – often refer to the profound impact of others on the responses of the participant. Consequently, this implies that THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 34 –

psychological research systematically labels one of the fundamental characteristics of the object it intends to investigate as bias (!).

Another pertinent characteristic of complex systems is historicity. In sharp contrast to mechanical systems, the behavior of a complex system depends on its entire historical trajectory.

Convection rolls, for instance, behave with a certain ‘historical awareness’, in the sense that the way in which the rolls structurally develop and manifest depends on all previous behaviors (Yin & Herfel, 2011, p. 407). This implies that descriptions of single aspects of complex systems inevitably have to refer to the non-local, historically developed, larger scale structures of the system (which also relate to the characteristic of hierarchy, i.e. the coherence between the parts and the whole of complex systems, Bishop, 2011, p. 114). However, ‘the entirety of the system’ is intrinsically multidimensional. It is composed of a variety of characteristics, which renders it unsuitable for metric description and necessitates narrative and/or mathematical description (Schmidt, 2011, p.

247). We have already argued from a network perspective that the psychic system – almost self-evidently – displays the characteristic of historicity. This can be observed in a very concrete way in the measurement context. For example, if you confront a participant with an item on a questionnaire that asks ‘Do you feel depressed?’, the response to this item will not only reflect ‘the amount of depression’ in this person, but also, to some extent, the participant’s entire historically formed network that comes into play to make possible the interpretation of that item (Rosenzweig, 1933, p.

337). One subject might interpret the term ‘depression’ in a differentiated and sophisticated way, because the subject was raised by parents who worked in mental health care, while for another subject this same term might only evoke a global image of psychological distress. We will describe in the subsequent chapters how

– 35 –

this problem has been recognized by some scholars and how some of them have attempted to develop quantitative methodologies to solve it (e.g. Steyer, Mayer, Geiser, & Cole, 2015). In accordance with the line of reasoning presented here, we nevertheless believe that any thorough scientific inquiry implies that at some point the multidimensional, historical entirety of the subject’s meaning-making apparatus is narratively described.

FIGure 2

Lorenz’s chaotic waterwheel (image retrieved from Leys, Ghys, & Alvarez, 2003).

tHe MeASureMent ProBLeM FroM A concePtuAL PerSPectIVe

– 36 –

60 50 Z-VALUES

The field of complex dynamical systems has yielded splendid examples of mathematical description. Perhaps the most illustrative is Edward Lorenz’s famous chaotic waterwheel. A chaotic waterwheel is a wheel that has buckets with a hole in the bottom fixed to it, with water pouring in at the top (Figure 2). When the initial volume of water is small, the wheel moves at a constant pace in the same direction. However, as the volume increases and reaches a critical limit, the wheel starts to behave chaotically; its speed becomes irregular and the direction of rotation changes in an unpredictable way. Lorenz noticed that in many respects the waterwheel behaved like convection rolls and he was able to describe the chaotic movement at each moment with a set of three non-linear equations with three unknown variables x-y-z.

Because the wheel behaves non-periodically, the values for x-y-z never stabilize and the same combination of values is never exactly repeated. In a moment of genius, Lorenz had the idea to plot the successive values of the three variables as x-y-z values in an orthogonal axis system. In an astonishing way, an esthetically magnificent (see also Ruelle, 1980, p. 137) butterfly picture appeared, which became known as the Lorenz attractor (Figure 3). In this way, Lorenz was able to visualize the order beneath a chaotic phenomenology by representing it in what later became known as phase space. Gleick (1987, p. 135) describes it like this: “Phase-space portraits of physical systems exposed patterns of motion that were otherwise invisible, as an infra-red landscape photograph can reveal patterns and details that exist just beyond the reach of perception.” This is also reminiscent of what Husserl (1907, p. 141) called ‘the pre-empirical’, which is a space that pre-exists empirical space and determines it. And as previously remarked above, it also has much in common with Plato’s universal forms that underlie the world as we experience it.

– 37 –

40 30 20 10 0 -10

-25 -20

-15

-10

-5

X-VALUES

0

5

10

15

-30 20 -40

-20

-10

0

10

20

30

Y-VALUES

F I G URE 3

The Lorenz attractor (image retrieved from Reed, 2014).

This framework can now be used to evaluate contemporary measurement practice with reference to the nomothetic and the idiographic paradigms in psychology.

The differences between the two approaches were well summarized by Watson (1934) in his paper in Psychological Bulletin.

The idiographic paradigm focuses on single case studies and considers measurements to be helpful, but only if they are situated in the context of the narrative descriptions of every single subject.

Consider in this respect the following quotation from William Stern (1921, pp. 3-4, cited in Lamiel, 2009, p. 72), the inventor of the Intelligence Quotient: “For the examinee in question, tests yield a number on the basis of which that examinee can be located somewhere along a quantitative scale, but which obscure things qualitatively peculiar to that individual. The results of direct observation cannot be quantified but make possible a qualitative refinement of the psychological profile. For all of these reasons, the methods of direct observation of an examinee must always be used to supplement the test methods, and the former must be developed with the same care THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 38 –

as the latter.” This basic idea was widely shared by major scholars before the Second World War, including leading behaviorists such as Watson, Skinner and Pavlov. The latter, for instance, claimed that the quantitative data on his salivating dogs were useless and could not be interpreted correctly if one was not familiar with the global character of every single dog (Todes, 2014).

The nomothetic paradigm investigates psychological processes in a completely different way. First, it relies almost exclusively on metric description; narrative description is considered less ‘objective’ and less ‘accurate’. Second, in order to control the subjectivity of the researcher, measurement procedures are standardized as strictly as possible. Third, to guarantee the generalizability of findings, measurement is applied to as large a sample as possible.

Fourth, since the global idiographic description of single subjects can only take place in a narrative way, it is always regarded as being subjective and therefore should be discounted. In other words, measurements are no longer interpreted within the context of the narrative data of single subjects, but in a standardized way, thereby presupposing that the same measurement outcome means the same things in different participants.

Before the Second World War, the nomothetic paradigm was dominant in America, whereas the idiographic paradigm held sway in the continental academic world, in particular in Germany and Austria. After 1945, the nomothetic paradigm quickly gained momentum and became dominant in the entire Western academic world. Importantly, this did not happen because it prevailed in intellectual debate, but because various levers of power crushed the idiographic paradigm. More specifically, the latter paradigm faded into the background because many of the German and Austrian scholars in psychology happened to be Jewish and

– 39 –

were expelled from their academic positions by the Nazi-regime (Watson, 1934), or else later met their fate in the camps.

In its attempt to quantify psychic phenomena, psychology has developed a plethora of measuring instruments, which can be divided into explicit and implicit measures. Examples of explicit measures are researcher rating scales, q-sorts and peer ratings applied to the participants’ own self-reported material, such as the verbal data from interviews, and the well-known self-report questionnaire. Explicit measures assess a psychological characteristic in a straightforward way. The test stimuli – for example, the questionnaire items – ask a participant to what extent he possesses a particular characteristic – for instance, by giving a score on a Likert scale. Explicit measures therefore rely on the participants’ introspective abilities and on their willingness to report the characteristic under investigation. Perhaps even more fundamentally, explicit measures always assume that participants will be able to attribute the meaning intended by the researcher to the test stimuli.

In terms of our network model, they assume that specific test stimuli will activate an identical, ‘intended-to-be-measured’ set of nodes and links in all the different participants (see Figure 4, left part).

THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 40 –

– 41 –

MEASUREMENT APPARATUS

TOTALITY OF THE MEASUREMENT SITUATION MEASUREMENT APPARATUS

S R G

O

F

K

F

L

A

N

D E

C B

M

I

R O

X

H

E

C B

R

R

N

D

A

S

G

X

H

S

S

K

L

I

F I G URE 4

Classical explicit and implicit measurement procedures start from the assumption that isolated test stimuli activate isolated aspects of the participants’ associative networks (left); logical-conceptual reasoning suggests the measurement procedures consist of a complex and variable set of stimuli that activate a complex and variable set of aspects of the associative network.

Examples of implicit measures are Affective Priming Methods (APM) and Implicit Association Tests (IAT). It is beyond the scope of this book to engage in a differentiated discussion of these measures. In general, we can say that rather than assessing a conscious aspect of psychic experience consisting of a set of links and nodes (as is the case with explicit measures), implicit measures assess one specific associative link in the participants’ networks.

They typically do this by asking participants to judge the nature of a relationship between two stimuli – usually isolated words or images – and by using the response time to infer something about the strength of the link. The response is typically provoked in an automatic way, which aims to avoid a series of biases that manifest themselves in explicit measures, such as the participants’ sensitivity to compliance, social desirability and impression management (e.g. Fazio, Sanbonmatsu, Powell, & Kardes, 1986, p. 237).

M

It may be useful at this point to introduce a definition of validity for the evaluation of these measures from the perspective of our logical-theoretical argumentation. Several definitions have been proposed, some of which, like the Cronbach and Meehl definition (1955), are grounded in the concept of construct validity. However, construct validity is now increasingly regarded as a problematical concept (e.g. Borsboom et al., 2004, p. 1061), since it is based on computing correlations with other measures that inevitably also show limitations at the level of validity. For this reason, in keeping with Kelley (1927), Cattell (1946) and Borsboom et al. (2004), I prefer to use the following definition: a psychological measure is valid to the extent it measures the psychological attribute it intends to measure, and, consequently, it lacks validity to the extent that it is influenced by factors not intended to be measured (also see Borsboom et al., 2004, p. 1061).

Both explicit and implicit measures actually start from the general assumption that specific test stimuli activate specific, intended-to-be-measured aspects of the network, which then leads to a specific response (Figure 4, left part). It should be clear that this assumption is by and large incompatible with the complex and dynamic nature of psychic phenomena. First, the idea of presenting a standardized and isolated set of specific test stimuli is per se already problematic. The totality of the test procedure always consists of a complex and never completely identical set of stimuli that all impact on the participant’s psychic system. As a consequence of the sensitivity of complex dynamical systems to initial conditions, it can be expected that even minimal differences in the testing procedures might generate radically different activation patterns. Second, the participants’ networks are far from identical, which further reduces the chance of activating identical, intended-to-be-measured characteristics in each of them.

THE MEASUREMENT PROBLEM FROM A CONCEPTUAL PERSPECTIVE

– 42 –

Thus, it is to be expected that the activation spreading throughout the network in response to the test stimuli will inevitably activate a complex and constantly varying set of nodes, including many factors not intended to be measured (Figure 4, right part). As a result, validity will be low.

Classical measures start from a rigid and mechanical conception of psychic processes, which from a conceptual point of view is now outdated. As Toomela (2007, p. 11) has put it: “For over more than half a century, the behaviorist thinking was replaced by cognitive psychology … Despite that not only modern mainstream cognitive psychologists, but also researchers in all other fields of psychology behave like behaviorists … measures are used as if a patient is an inert subject. It is worth mentioning that the idiographic paradigm, except in exceptional cases where it works with idiosyncratic items, also starts from the principle of measuring specific characteristics by means of specific, standardized stimuli. Purely at that level, it therefore suffers from comparable shortcomings.

However, it can be distinguished positively from the nomothetic paradigm at the level of interpreting measurements, which always occurs in an idiosyncratic way that is more in keeping with the complex and dynamic nature of the psychic system.