The statistical argument against alternative medicine: why it “cannot be considered evidence-based”

Most of the underlying assumptions of alternative medicine (AM) lack plausibility. Whenever this is the case, so the argument put forward by an international team of researchers in a recent paper, there are difficulties involved in obtaining a valid statistical significance in clinical studies.

Using a mostly statistical approach, they argue that, since the prior probability of a research hypothesis is directly related to its scientific plausibility, the commonly used frequentist statistics, which do not account for this probability, are unsuitable for studies exploring matters in various degree disconnected from science. Any statistical significance obtained in this field should be considered with great caution and may be better applied to more plausible hypotheses (like placebo effect) than the specific efficacy of the intervention.

The researchers conclude that, since achieving meaningful statistical significance is an essential step in the validation of medical interventions, AM practices, producing only outcomes inherently resistant to statistical validation, appear not to belong to modern evidence-based medicine.

To emphasize their arguments, the researchers make the following additional points:

It is often forgotten that frequentist statistics, commonly used in clinical trials, provides only indirect evidence in support of the hypothesis examined.

The p-value inherently tends to exaggerate the support for the hypothesis tested, especially if the scientific plausibility of the hypothesis is low.

When the rationale for a clinical intervention is disconnected from the basic principles of science, as in case of complementary alternative medicines, any positive result obtained in clinical studies is more reasonably ascribable to hypotheses (generally to placebo effect) other than the hypothesis on trial, which commonly is the specific efficacy of the intervention.

Since meaningful statistical significance as a rule is an essential step to validation of a medical intervention, complementary alternative medicine cannot be considered evidence-based.

Further explanations can be found in the discussion of the article where the authors argue that the quality of the hypothesis tested should be consistent with sound logic and science and therefore have a reasonable prior probability of being correct. As a rule of thumb, assuming a “neutral” attitude towards the null hypothesis (odds = 1:1), a p-value of 0.01 or, better, 0.001 should suffice to give a satisfactory posterior probability of 0.035 and 0.005 respectively.

In the area of AM, hypotheses often are entirely inconsistent with logic and frequently fly in the face of science. Four examples can demonstrate this instantly and sufficiently, I think:

Homeopathic remedies which contain not a single ‘active’ molecule are not likely to generate biological effects.

Healing ‘energy’ of Reiki masters has no basis in science.

Meridians of acupuncture are pure imagination.

Chiropractic subluxation have never been shown to exist.

Positive results from clinical trials of implausible forms of AM are thus either due to chance, bias or must be attributed to more credible causes such as the placebo effect. Since the achievement of meaningful statistical significance is an essential step in the validation of medical interventions, unless some authentic scientific support to AM is provided, one has to conclude that AM cannot be considered as evidence-based.

Such arguments are by no means new; they have been voiced over and over again. Essentially, they amount to the old adage: IF YOU CLAIM THAT YOU HAVE A CAT IN YOUR GARDEN, A SIMPLE PICTURE MAY SUFFICE. IF YOU CLAIM THERE IS A UNICORN IN YOUR GARDEN, YOU NEED SOMETHING MORE CONVINCING. An extraordinary claim requires an extraordinary proof! Put into the context of the current discussion about AM, this means that the usual level of clinical evidence is likely to be very misleading as long as it totally neglects the biological plausibility of the prior hypothesis.

Proponents of AM do not like to hear such arguments. They usually insist on what we might call a ‘level playing field’ and fail to see why their assumptions require not only a higher level of evidence but also a reasonable scientific hypothesis. They forget that the playing field is not even to start with; to understand the situation better, they should read this excellent article. Perhaps its elegant statistical approach will convince them – but I would not hold my breath.