Psychology and many related fields are in the midst of what can be viewed as a coming-of-age crisis. Following a stream of depressing revelations about a lack of reliability in the field, lots of researchers are dedicating themselves to improving the discipline’s rigor.

The latest proposal to up that rigor is a big one: 72 researchers from a range of disciplines have drafted a manuscript arguing that the threshold for claiming “statistical significance” should become much stricter. There’s often a fair amount of consensus on how science could be improved, but this suggestion has stimulated some intense debate.

Statistical significance in a very small nutshell

Statistical significance is a concept underlying a huge amount of science—not just psychology or social sciences, but medicine, life sciences, and physical sciences, too. “Significance” used in this way doesn’t mean the importance or size of a finding, rather it's the probability of that finding showing up in your data even though your hypothesis turns out to be wrong.

If you have a hypothesis—say, reiki helps pain management—you’ll get two equivalent groups of people with pain problems, give one group reiki, and the other group some kind of placebo treatment. Then you’ll see how they all feel after the treatment and if there’s a difference between the two groups. If you’re right about your hypothesis, you should see a difference between the groups. If you’re wrong, and reiki doesn’t help, there shouldn’t be a difference.

The problem is that people are variable, so there will always be some degree of difference between two groups. What we want to know is the following: if you’re wrong, and reiki doesn’t help with pain, how likely is it that you’d be seeing the difference you get in your data? If reiki is useless, it would be extremely unlikely that you’d see a big difference between the groups, so it would be fair to take a big difference as evidence that reiki does help.

The problem is that sometimes, you'll get a big difference even if reiki is useless—that’s just how probability and human variability work.

Currently, many fields have the same threshold for acceptable evidence: you’d be able to use your study as evidence if you calculated that your data would have less than a 5-percent chance of occurring if reiki were useless. That 5-percent threshold is the p-value: if p is less than 0.05, that's statistically significant. If it's larger, it's not.

The suggestion

The 5-percent threshold was always an arbitrary cut-off, proposed by statistician Ronald Fisher in 1925. A chance of 5.1 percent is not objectively more meaningful than a chance of 4.9 percent. Fisher acknowledged that it was arbitrary, and he emphasized that it should be used carefully and in conjunction with other information to reach a sensible conclusion. There are already fields (like physics and genetics) that have different thresholds.

In a paper, which is due to appear in Nature Human Behavior, behavioral economist Daniel J. Benjamin and his 71 co-authors say that a 5-percent chance is too high, and they propose lowering the threshold to 0.5 percent. Values between 0.5 percent and 5 percent could be labeled as “suggestive.” There’s a litany of problems affecting science, but the paper's authors argue that even if everyone were doing flawless science in a flawless science publication system, the statistical threshold would still be a problem.

The benefits, write Benjamin and colleagues, would be meaningful: a lower threshold would reduce the number of false positives, making it more likely that any significant finding could be replicated by other researchers. It should also push people toward using larger sample sizes, which would help to limit some of the problems that come with studying small groups. Ideally, that would all lead to a more robust and reliable body of research.

The objections

The p-value is far too often treated as the golden ticket of science. Finding a significant effect has long meant being able to get your work published, while a non-significant effect meant results that remained buried in a file drawer. This has meant not just that important null results don’t see the light of day (wouldn’t it be important to tell people if that reiki experiment found no effect?), but also that the p-value is subject to all kinds of abuse as people try to find ways of presenting their results as publishable.

Plenty of scientists have taken to blogs and Twitter to discuss flaws in the threshold-changing plan. One common objection is that the whole statistical school of thought underlying the use of p-values is riddled with problems and that all efforts should be focused on encouraging people to use better methods.

Another argument is that this proposal won’t do anything to change misguided use of p-values—and it might actually entrench bad behaviors. A lower threshold isn’t going to stop people treating significance testing as the be-all and end-all of their research, letting the p-value do the thinking for them. “Statistical thresholds are not up to the job we so badly want them to do,” writes psychologist David Funder on his blog.

The researchers suggesting the change don’t think it will even come close to fixing all the problems in science, but they see it as one of many necessary changes to the system—and a quick and pragmatic one to implement. “You have a leaky pipe. Do you patch it up until you can get a plumber round to fix it properly or leave water dripping all over your floor?” tweeted Andy Field, one of the 72 co-authors.

But others think that this change, without other structural fixes being in place first, “may even do significant harm,” writes computer scientist Shlomo Argamon on his blog. Lowering the false positive rate without addressing other fundamental problems “could lead to a false sense of security and entrench bad methodological attitudes,” he argues.

It’s not a debate that’s likely to end any time soon, because the consequences of the proposed change are difficult to predict. As neuroscientist Beau Sievers tweeted in response to Field, “the question is not 'should we fix it?'—yes, we should fix it—but 'will fixing it this way break something else?'”

PsyArXiv, 2016. DOI: 10.17605/OSF.IO/MKY9J (About DOIs).