Ben Goldacre, The Guardian, Saturday 23 April 2011

Last year a mainstream psychology researcher called Daryl Bem published a competent academic paper, in a well respected journal, showing evidence of precognition. Instead of designing new studies to see whether people could consciously tell you about the future, he ran some classic psychology experiments backwards.



For example, in experiments on subliminal influence, participants are presented with two mirror images of the same picture, asked which they prefer, and are likely to choose the images where a subliminal negative image is flashed up for milliseconds, before they make their choice. In the Bem study, the negative images were flashed up after they made their choice, but participants were still less likely to choose the image on the side with the nasty subliminal image.

This was all pretty kosher, and statistically significant, and I wasn’t very interested, for the same reasons you weren’t: if humans really could see the future, we’d probably know about it already; and extraordinary claims require extraordinary evidence, rather than one-off findings. There’s plenty of amazing stuff, in our infinitely distracting universe, and I’ll pay attention to the cheesy precognition stuff when the evidence is good and replicated.

Now the study has been replicated. Three academics – Stuart Richie, Chris French, and Richard Wiseman – have re-run three of these backwards experiments, just as Bem ran them, and found no evidence of precognition. They submitted their negative results to the Journal of Personality and Social Psychology, which published Bem’s paper last year, and the journal rejected their paper out of hand. We never, they explained, publish studies that replicate other work.

This squabble illustrates two problems facing all of science, which have still never been adequately addressed.

The first is the problem of context: these positive results may have happened purely by chance, against a backdrop of negative results that never reached the light of day. We know that researchers and academic journals, just like newspaper journalists, are more likely to publish eyecatching positive results. We know that even if you analyse one study’s results in lots of different ways, you increase the likelihood of getting a positive finding purely by chance.

So replicating these findings was key – Bem himself said so in his paper – and keeping track of the negative replications is vital too. For clinical trials, there is a system of registering your trial before you recruit participants, to reduce the risk of negative results being buried (it’s imperfect, as I’ve written, but it exists). Outside of trials, people tend not to bother, which puts whole fields at risk of spurious positive findings: Wiseman has set up a register for people to declare that they were attempting to replicate Bem’s work.

But the second issue is how people find out about stuff. We exist in a blizzard of information, and stuff goes missing: as we saw recently, research shows that people don’t even hear about retractions of outright fraudulent work. Publishing a follow-up in the same venue that made an initial claim is one way of addressing this problem (and when the journal Science rejected the replication paper, even they said “your results would be better received and appreciated by the audience of the journal where the Daryl Bem research was published”).

The same can be said for the New York Times, who ran a nice long piece on the original precognition finding, New Scientist who covered it twice, the Guardian who joined in online, the Telegraph who wrote about it three times over, New York Magazine, and so on.

It’s hard to picture many of these outlets giving equal prominence to the new negative findings that are now emerging, in the same way that newspapers so often fail to return to a debunked scare, or a not-guilty verdict after reporting the juicy witness statements.

All the most interesting problems around information today are about structure: how to cope with the overload, and find sense in the data. For some eyecatching precognition research, this stuff probably doesn’t matter. What’s interesting is that the information architectures of medicine, academia and popular culture are all broken in the exact same way.