My previous post on publication bias only touches the surface of our psychological and societal biases. I find publication bias particularly pernicious because it is a phenomenon sitting right at the centre of social science and could be largely avoided if we wanted to, but apparently we don’t. The attention that this meta issue receives is negligible, the institutional bulwarks of social science ignore it, and the disconnect between the crucial but glossed over distortion of what science tells us that is caused by this bias and the nitpicking issues that are at the heart of what very public science wars are fought over seems so great that it is depressing.
But publication bias is only one of the contributing factors that make the social science output that is available in the public domain so dodgy. It makes for a published corpus of results that is near totally about providing substantiating evidence that particular theoretical ideas might be true (yes it’s laborious to avoid the word confirmation here, but that’s very much on purpose). All the work that has tried to find such evidence but wasn’t successful is not submitted for publication, because it is not considered publishable, or if it was, it was indeed not considered publishable by the journal editors and peer reviewers who overwhelmingly dislike “null results”.
In practice, researchers, reviewers and editors all disregards that science is not about confirmation (told you so) but about falsification (Popper‘s Logic of Scientific Discovery). Weird because theoretically most would subscribe to that view and would feel insulted if the genuineness of their belief would be doubted.
Before continuing, let’s lighten up a bit with Mr. Kite. The Across the Universe version that I find an improvement upon the original, quite a feat.
My fascination with Nassim Taleb‘s work (see also my reads page) is partially because of the many ways he shows the ridiculous and disastrous consequences of focusing on finding something and equating not finding with non-existence. This is statistical nonsense but social science academia is totally infused with it.
Publication bias is an end-of-the-pipeline phenomenon. Earlier in the production process the obsession with positive results creates many more false positive results than the statistical legitimation accounts for. An illustrating but technical example of how such false positives may arise, given the uncontrolled degrees of freedom of researchers, is this article in the online journal Psychological Science. Again, an aberration that would be relatively easy to (at least partially) repair, if the scientific community would be genuinely interested. What should be noted is that one doesn’t have to assume malevolence, the statistical ignorance of most social scientists is more than enough. However, what beats me is, that once we know this, why does business continue as usual?
However that may be, these two don’t exhaust the biases responsible for our distorted social science view of the world. What worries me is that the level of distortion arising from this way of looking at my profession is crippling. A simplistic metric would be to add up the bias due to what ends up in the published corpus (huge) and the bias due to many more false positives than the research reports (potentially huge) and the bias due to other factors, and given that this would be an underestimate, we end up with, yes indeed, what do we end up with?
Underlying all if this is our predilection to confirmation bias. That’s hard-wired into our cognitive system. Cannot escape it. So social science in my book is all about working our way around this bias, inoculating our inferential processes against it. We’re not doing a good job at it….
Why? For one, because being in academia doesn’t switch off one’s very human preferences. Falsification goes against the grain of those. We’ve established a probably truth, our instinct makes us see what confirms, but our profession asks us to throw all we have at undermining it? Bad luck for the profession. For another, because academia is an institutional stage with all kinds of perverse incentives for its actors to not rock the boat.
I admit not answering the title question. To a certain extent my blog is all about finding that answer, so consider it a work in progress, most probably to remain unfinished.
To end on a positive note, at least for those who are Dutch and of my generation, enjoy this bit of visual and auditory nostalgia: