A summary of the evidence that most published research is false

One of the hottest topics in science has two main conclusions:

Most published research is false

There is a reproducibility crisis in science

The first claim is often stated in a slightly different way: that most results of scientific experiments do not replicate. I recently got caught up in this debate and I frequently get asked about it.

So I thought I’d do a very brief review of the reported evidence for the two perceived crises. An important point is all of the scientists below have made the best effort they can to tackle a fairly complicated problem and this is early days in the study of science-wise false discovery rates. But the take home message is that there is currently no definitive evidence one way or another about whether most results are false.

I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely

We need more statistical literacy

We need more computational literacy

We need to require code be published

We need mechanisms of peer review that deal with code

We need a culture that doesn’t use reproducibility as a weapon

We need increased transparency in review and evaluation of papers

Some of these have simple fixes (more statistics courses, publishing code) some are much, much harder (changing publication/review culture).

The Many Labs project (Paper 8) points out that statistical research is proceeding in a fairly reasonable fashion. Some effects are overestimated in individual studies, some are underestimated, and some are just about right. Regardless, no single study should stand alone as the last word about an important scientific issue. It obviously won’t be possible to replicate every study as intensely as those in the Many Labs project, but this is a reassuring piece of evidence that things aren’t as bad as some paper titles and headlines may make it seem.

The Many Labs results suggest that the hype about the failures of science are, at the very least, premature. I think an equally important idea is that science has pretty much always worked with some number of false positive and irreplicable studies. This was beautifully described by Jared Horvath in this blog post from the Economist. I think the take home message is that regardless of the rate of false discoveries, the scientific process has led to amazing and life-altering discoveries.

Please enable JavaScript to view the comments powered by Disqus.

Disqus