(Thomas Peter/Reuters)

Research findings that can’t be replicated may end up stalling medical progress.

Editor’s Note: Pascal-Emmanuel Gobry, a conservative writer and a fellow at the Ethics and Public Policy Center, is writing a series of columns on uncomfortable truths about health care in America. Some will make conservatives more uncomfortable, others will make progressives more uncomfortable, but most should make everyone uncomfortable.

Everybody agrees the U.S. health-care system is broken. They disagree on why, on what’s wrong, and on how to fix it; but they agree the system is broken. Yet while we might be paying too much for it, we might be applying it in wrong and wasteful and destructive ways, and we might be slowing it down, we still produce steady improvements in the quality of medicine and care.


But what if that’s not the case? What if even the science behind the health-care system is stalling?

Many scientific fields are facing a “replication crisis.” The modern scientific method is based on the idea that experiments can be replicated. The point of a scientific experiment is to isolate every possible cause from an experiment except the one under study, so that we can tease out the cause-and-effect relationships that go into the phenomena being studied. This means that a similar experiment conducted by different people should yield the same result. If a given study cannot be replicated, then the likelihood is very high that it is a fluke.

The replication crisis in experimental psychology has been well publicized. But there is a replication crisis going on in biomedicine at the moment too, and it is even worse. The rule of thumb in psychology is that around half of the experimental results cannot be replicated, but as the engineer and writer Will Wilson points out in an important article on the state of science, a landmark 2011 study from a team at Bayer trying to replicate seemingly important findings in pre-clinical cancer biology failed to replicate 75 percent of them. In another study, only 11 percent of the results could be replicated.



And it gets worse: The studies that they had failed to replicate were on average cited more often than the ones that worked. “Some non-reproducible preclinical papers had spawned an entire field,” the authors wrote, “with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis.” In other words, a lot of the basic science that is used every day to do cancer research or discover new drugs is essentially junk.

According to Glenn Begley, a former head of cancer research at Amgen who decried the state of biomedical research in a bombshell 2012 article in Nature, pre-clinical research — the basic science, most often done in government or government-funded labs, on which the rest of biomedical science is based — is often done according to bad standards, with scientists failing to use the proper controls and procedures.

Systematically attempting to replicate new findings in biomedicine — something that should be standard — is an enormously expensive and time-consuming proposition if it can be done at all.


As New America’s Future Tense project noted, an additional problem in biomedicine is that replicating experiments is expensive and complicated. For most experiments in psychology, you need a dozen college undergraduates as guinea pigs and a room. A biomedical experiment involves very expensive equipment and scarce laboratory resources, as well as the red tape that comes with handling sensitive material such as cancer cells and DNA samples. Systematically attempting to replicate new findings in biomedicine — something that should be standard — is an enormously expensive and time-consuming proposition if it can be done at all: Sometimes experiments involve material that was destroyed or degraded, or enormously complex protocols that cannot be replicated because the experiment was done years ago and the people doing it did not write down everything they did minute by minute.


In other words, it’s not just the financing and delivery of health care that’s broken, but also the science behind it. And the problem is systemic. It’s one thing for some scientific challenges to be very difficult, so that progress is slow, grinding, and frustrating. But replicability is a foundation of the scientific method. Without it, the problem is not that scientists are tackling a hard problem, it’s that they don’t have the tools to even be sure they’re doing science at all.

And yet we need the science to work. Perhaps the most terrifying story in science over these past few years is antibiotic resistance. After centuries of humans’ attacking bacteria with antibiotics, the bacteria have evolved to be increasingly resistant to them. For all the wonderful progress in things like organ transplants and cancer treatments, the vast majority of the progress in life expectancy and quality that medicine has brought about over the past 200 years has been due to antibiotics. If our antibiotics stop working, for all our whiz-bang medical equipment, we would be completely at the mercy of global pandemics, and child mortality would become an everyday reality, as infections that today are mild and easily treated would become life-threatening again.


It’s possible that even if we could implement the perfect health-care-reform agenda tomorrow (that is, if we had any idea what it was), health care would still stall, or even get dramatically worse, because the underlying science has broken.

READ MORE:

There are No Good Options On Health Care

A Culture that Fears Death

The Most Wasteful (and Popular) Health Spending

— Pascal-Emmanuel Gobry is a fellow at the Ethics and Public Policy Center.