Editors at a respected scientific journal are reconsidering their decision to publish a study, which claims that a homeopathic dilution of poison oak can reduce pain in rats, after online critics pointed out that the study is rife with bogus, sloppy, and low-quality data.

The study—titled “Ultra-diluted Toxicodendron pubescens attenuates pro-inflammatory cytokines and ROS-mediated neuropathic pain in rats”—was published September 10 in Scientific Reports, an open-access journal run by the Nature Publishing Group. Now, the online manuscript runs with an editor’s note at the bottom, stating:

Readers are alerted that the conclusions of this paper are subject to criticisms that are being considered by the editors. Appropriate editorial action will be taken once this matter is resolved.

The criticisms began mounting last month as independent scientists—rightly skeptical of homeopathy’s pseudoscientific principles—took a closer look at the paper.

As Ars has reported before, homeopathy is based on the idea that toxic substances that can produce symptoms similar to a given ailment can cure that ailment (“like cures like) when excessively diluted (“law of infinitesimals”). Resulting dilutions often contain no remaining molecules of the original substance or very trace amounts. Some homeopaths believe that water molecules can have memories of the substance and that shaking the dilutions can increase their healing potency.

The study is far from the first example of a peer-reviewed journal giving a pass to a shoddy study purporting to prove the efficacy of homeopathic products (as Ars has long noted). However, the fact that the study survived peer-review at such a high-profile, well-regarded journal was particularly alarming to members of the scientific community.

“It’s worrying that a major journal like Scientific Reports didn’t pay close attention to a study that claims to show that homeopathy works," Enrico Bucci told Nature News (the outlet reports that it works independently of its publisher Springer Nature, which also published Scientific Reports). Bucci is a systems biology researcher at Temple University in Philadelphia, Pennsylvania, who also carried out some of the analysis finding egregious flaws in the homeopathy study.

In all, scientists including Bucci and commenters on PubPeer noted that the study as published used duplicated data figures that claimed to show different experimental results, inconsistently reported data and results for various treatment dilutions in the text and figures, contained suspiciously identical data points throughout a series of figures that were reported to represent different experimental results, and hinged on subjective, non-blinded data from a pain experiment involving just eight rats.

Specifically, Figure 1 of the study is uninterpretable. The study authors claim that the data in this figure shows that ultra-dilutions of poison oak (abbreviated RT for RhusTox) mitigates oxidative stress in cell cultures. However, the text discusses the results for dilutions of 1×10−8, 10−12, 10−24, and 10−30, while the corresponding (Figures 1B, 1C, and 1D) data show results for weaker dilutions: 1×10-2, 10-4, 10-6, and 10-8.

Also, Figure 1G and 1H show the exact same data plot duplicated, labeled identically in the graphs but differently in the text of the figure legend. Figures 1I and 1J show another duplicated data figure, but the copies are labeled differently in both the graph and the legend text, incorrectly suggesting the duplicates show data from two different experiments.

Similarly, in Figure 2, which the authors claim indicates that the ultra-diluted homeopathic treatment has anti-inflammatory properties in cells, the dilution discussed in the text is different from that of the labeled data presented in the figures.

Researchers note that Figure 3 of the paper appeared to include suspiciously identical data points across different experiments. The authors report that the data presented in Figure 3 claims to show that eight rats delayed their paw withdraw from painful warm, cold, and mechanical probing treatments after being treated with either a control pain relief drug or a poison oak dilution. But many of the data points in the graphs of the three pain treatments are identical, Bucci points out. “ This coincidence is impossible ,” he wrote in a blog post criticizing the study.

Lastly, others pointed out that even if the data is somehow accurate, the experiment is unconvincing. The fast timing differences of paw withdraw is subjective. It’s also prone to bias because the researchers were not blinded to the rats’ treatments (meaning they could have known which animals were given the control drug or the homeopathic dilution). Moreover, eight animals in each group is not a large enough number from which to draw firm conclusions, they argue.

In comments to Nature News, a senior author of the study—pharmacologist Chandragouda Patil of the R.C. Patel Institute of Pharmaceutical Education and Research in Dhule, India—acknowledged that there were errors in the study. But Patil asserted that they were simply typos and that “this does not change the scientific conclusions in any way.” He added that the study was done “with utmost integrity.”

Patil said that he and the other authors would request that Scientific Reports simply update the article with the typos corrected.

Scientific Reports, 2018. DOI: 10.1038/s41598-018-31971-9 (About DOIs).