Small studies with low power undermine the reliability of science and new evidence suggests that low power is the norm in neuroscience

When I started my PhD a few years ago, I thought that certain psychological findings were established fact. The next four years were an exercise in disillusionment. If the effects I was seeking to explore were so reliable, so established, why could I not detect them?

There is growing interest in the need to improve reliability in science. Many drugs show promise at the design and pre-clinical phases, only to fail (at great expense) in clinical trials. Many of the most hyped scientific discoveries eventually cannot be replicated.

Worryingly for science (but somewhat comforting for my self-esteem as a researcher) this may be because many of the conclusions drawn from published research findings are false.

A major factor that influences the reliability of science is statistical power. We cannot measure everyone or everything, so we take samples and use statistical inference to determine the probability that the results we observe in our sample reflect some underlying scientific truth. Statistical power determines whether we accurately conclude if there is an effect or not.

Statistical power is the ability of a study to detect an effect (eg higher rates of cancer in smokers) given that an effect actually exists (smoking actually is associated with increased risk of cancer). Power is related to the size of the study sample (the number of smokers and non-smokers we test) and the size of the real effect (the magnitude of the increased risk associated with smoking). Larger studies have more power and can detect smaller, more subtle effects. Small studies have lower power and can only detect larger effects reliably.

In a paper published today in Nature Reviews Neuroscience we reviewed the power of studies in the neuroscience literature, and found that, on average, it is very low – around 20%. Low power undermines the reliability of neuroscience research in several important ways.

False negatives

Low power reduces the chances of detecting true effects. If there are 100 true effects to be discovered, studies with 20% power on average would be expected to only discover 20 of them. As the scientific mantra goes, absence of evidence is not evidence of absence. Therefore, the majority of findings from low powered studies are likely to be inconclusive and thus a waste of time and resources.

False positives

Perhaps less intuitively, when small low-powered studies do claim a discovery, that discovery is more likely to be false.

The probability of a research finding being true is related to the pre-study odds of that finding being true. These odds are higher for confirmatory or replication studies testing pre-specified hypotheses, as these have the weight of previous evidence or theory behind them. The odds are lower for exploratory studies that make no prior predictions, leaving the findings more open to chance.

The impact of combining low power and low pre-study odds has important consequences for the likelihood that the research finding is actually true. In our analysis, we show that for exploratory studies with an average 20% power, together with average one in four pre-study odds that the effect being sought is actually true, the likelihood that any claimed effect (based on passing a conventional level of statistical significance) actually is true is only 50%. That's a 50/50 chance that any positive effects are spurious. For a confirmatory study with four to one pre-study odds, the chance that any positive effects are spurious is reduced to 25%.

Winner's curse

When small, low-powered studies are lucky enough to discover a true effect, they are more likely to exaggerate the size of that effect.

Smaller studies are more susceptible to random variation between individuals than larger ones are, and can therefore only detect large effects. This is because the increased random variability makes it difficult to assess whether a small or moderate effect is due to random error or a true effect. Studies testing the same hypothesis will tend to find results that match the underlying true effect, but there will be some variation in their results due to differences between studies (different participants, researchers, settings, and so on). Some studies will overestimate the size of the association, and some will underestimate it, due to the chance fluctuations.

Small studies testing for an effect that is of moderate strength will mostly be inconclusive, because moderate effects are too small to detect with a small study. But, by chance, a few studies will overestimate the size of the association, observe an apparently large effect, and thus pass the threshold for discovery. This is known as the "Winner's curse" as the lucky scientist who finds evidence for an effect using a small study is often cursed to have found an inflated effect by chance.

Vibration effects

Just as small studies are more susceptible to random variation between individuals, they are also more susceptible to variability in research practice.

During a study, researchers make numerous decisions about which things to measure, how to analyse the data, which participants to include, each of which can nudge the results in this direction or that. The accumulation of such nudges in small studies can lead to dramatically different conclusions.

Consider, for example, excluding 10 participants from the analysis because, upon reflection, you thought they did not complete the experiment correctly. In a study of 20 people, this could drastically change the results, whereas a study of 2,000 would probably be relatively unaffected.

Publication bias: the file-drawer effect

There is a strong bias in science in favour of publishing novel, clear-cut, exciting findings and against publishing null-effects and replications. Editors are more likely to publish positive results, so researchers often file negative (i.e., null) findings in a drawer and forget about them. We've seen that small studies can only ever detect large effects and that low power increases the likelihood of spurious or chance findings, especially in exploratory studies. Couple this with a publication bias in favour of large, novel effects, and the implications for the reliability of the research literature is clear.

Implications

The current reliance on small, low-powered studies is wasteful and inefficient, and it undermines the ability of neuroscience to gain genuine insight into brain function and behaviour. It takes longer for studies to converge on the true effect, and litters the research literature with bogus or misleading results.

The preference for novel, exploratory research over solid, evidence-building replications exacerbates these problems. Replication is fundamental to good science, strengthening the signal of true effects against the backdrop of random noise. Given the disincentives to replication, spurious chance findings may never be refuted and may continue to contaminate the literature. More dangerously, unreliable findings may lead to unhelpful applications of science in clinical or community settings.

There is also an impact on young researchers. It can be demoralising to spend months trying to replicate a seemingly well-established effect to no avail, only to later find out that neither could many other researchers but they just didn't publish their data.

Reasons to be optimistic

There are reasons to be optimistic. Awareness of these issues is growing and acknowledging the problem is the first step to improving current practices and identifying solutions. Although issues of publication bias are difficult to solve overnight, researchers can improve the reliability of their research by adopting well-established (but often ignored) scientific principles:

We can consider statistical power in the design of our studies, and in the interpretation of our results. We can increase the honesty with which we disclose our methods and results. We could make our study protocols, and analysis plans, and even our data, publicly available. We could work collaboratively to pool resources and increase our sample sizes and power to replicate findings.

Kate Button is a research psychologist at the University of Bristol