‘Pilot studies’ of alternative medicine: incompetent, unethical, misleading and harmful – Apr 4 – by Edzard Ernst

During the last 25 years, my research focussed on the critical evaluation of all aspects of alternative medicine. I do not aim to promote this or that therapy or profession, my goal is to provide objective evidence and reliable information. I have noticed a proliferation of so-called pilot studies of alternative therapies.

In today’s anti-opioid climate, any non-opioid “treatment” for pain, is accepted uncritically, even if it’s only slightly effective for a few people, is promoted as “fighting the opioid crisis” (which is actually an “addiction crisis”)

A pilot study (also called feasibility study) is defined as a small scale preliminary study conducted in order to evaluate feasibility, time, cost, adverse events, and improve upon the study design prior to performance of a full-scale research project. Here I submit that most of the pilot studies of alternative therapies are, in fact, bogus. To qualify as a pilot study, an investigation needs to have an aim that is in line with the above-mentioned definition. Another obvious hallmark must be that its conclusions are in line with this aim. We do not need to conduct much research to find that even these two elementary preconditions are not fulfilled by the plethora of pilot studies that are currently being published, and that proper pilot studies of alternative medicine are very rare.

I have found that studies of alternative medicine for any kind of pain tend to exaggerate their positive results, even if they had to selectively fish through the data and subject it to torturous statistical methods to achieve them.

So, what are they, and why are they so popular in alternative medicine?

This article specifically concerns “pilot studies”, which in normal research do not generate results beyond establishing the feasibility of the research method examined.

The way I see it, they are the result of amateur researchers conducting pseudo-research for publication in lamentable journals in an attempt to promote their pet therapies (I have yet to find such a study that reports a negative finding). The sequence of events that lead to the publication of such pilot studies is usually as follows:

The steps below sound sadly correct and are corroborated by others who became disillusioned with this scam.

An enthusiast or a team of enthusiasts of alternative medicine decide that they will do some research.

They have no or very little know-how in conducting a clinical trial.

They nevertheless feel that such a study would be nice as it promotes both their careers and their pet therapy .

They design some sort of a plan and start recruiting patients for their trial.

At this point t hey notice that things are not as easy as they had imagined.

They have too few funds and too little time to do anything properly.

This does, however, not stop them to continue.

The trial progresses slowly, and patient numbers remain low.

After a while the would-be researchers get fed up and decide that their study has enough patients to stop the trial.

They improvise some statistical analyses with their results.

They write up the results the best they can.

They submit it for publication in a 3rd class journal and, in order to get it accepted, they call it a ‘pilot study’.

They feel that this title is an excuse for even the most obvious flaws in their work.

The journal’s reviewers and editors are all proponents of alternative medicine who welcome any study that seems to confirm their belief.

Thus the study does get published despite the fact that it is worthless. these studies pollute the medical literature and misguide people who are unable or unwilling to look behind the smoke-screen. Enthusiasts of alternative medicine popularise these bogus trials while hiding the fact that their results are unreliable. Journalists report about them, and many consumers assume they are being told the truth – after all, it was published in a ‘peer-reviewed’ medical journal! My conclusions are as simple as they are severe: Such pilot studies are the result of gross incompetence on many levels (researchers, funders, ethics committees, reviewers, journal editors).

They can cause considerable harm , because they mislead many people.

In more than one way, they represent a violation of medical ethics.

The could be considered scientific misconduct.

We should think of stopping this increasingly common form of scientific misconduct Author: Edzard Ernst MD, PhD , Emeritus Professor, Exeter University

During the last 25 years, my research focussed on the critical evaluation of all aspects of alternative medicine. I do not aim to promote this or that therapy or profession, my goal is to provide objective evidence and reliable information. This ambition did not endear me to many believers in alternative medicine.

I completely agree with Dr. Ernst: the studies of alternative medicine do not follow the rigorous standards applied to research on other medical treatments.

I have used many kinds of alternative therapies myself over the years and I do believe they hold value for some individuals some of the time.

However, they are not comparable to most other scientific research which is usually conducted with rigorous attention to proper scientific rules.