EDITOR'S NOTE: The paper this report was based on has been retracted, due to the journal editors deciding the results were not reproducible. That decision is related to some of the issues we discussed below.

I have great respect for medical professionals. Their training allows them to guess the identity of an affliction based on a set of vaguely expressed symptoms, and I'm amazed they do better than random chance.

Unfortunately, diagnosis is only half the job. Treatment is sometimes an exercise in frustration, especially for psychiatric problems. Issues like depression and schizophrenia, seem (to me) to be so poorly understood that the best doctors can do is to treat the symptoms and hope for the best. Combine that with the placebo effect and you have a recipe that tells us that as long as the doctor does something, things might get better. This idea has now been shown (again) in a study published in PLOS One.

In this clever study, doctors worked with a group of women suffering from depression (they were all either going through menopause or were post-menopausal). At the beginning of the trial, all of the women spent an hour or so in the company of the prescribing doctor, discussing their personal life and their symptoms and getting a thorough assessment of their mental state. The women were then randomly divided into three groups: one received a drug treatment while the other two were given placebos. At the beginning of the trial, four weeks in, and at the end (six weeks), all of the women were given a full battery of tests to quantify their depression.

During the study period, all of the women received weekly (I originally assumed that there was a weekly assessment, but this may not have occurred) monitoring for serious changes in symptoms.

Edit: the wording of the paper leaves a lot of room for doubt about how identically the three groups of patients were treated. In my opinion, the study blinding and protocol meant that the three groups were inadvertently treated differently by the medical professionals involved. In the original article, I wrote "But one of the two placebo groups received far more thorough monitoring. They essentially went through a large part of the first assessment every week, and the prescribing doctor would change either the placebo (to another placebo) or the dose of the placebo in response to any concerns." Although the paper states that treatment could change, it does not report the numbers of patients that had their medication changed. As earlier, I guessed that the assessment period was weekly, because it was not reported in the paper.

So how did this all turn out? Very well, in fact. It turns out that receiving the undivided attention of a medical professional for an hour a week does wonders for your mental health. The placebo/medical attention group showed far more improvement than both the drug and the other placebo group.

In the UK—which generally has pretty good health care—doctors do not have much time to spend with patients (maybe 15 minutes). For something like an infection, a cold, or something like that, this time limit is probably fine. But for mental health problems, I don't think there is any way that a patient can feel the doctor is taking a problem seriously in a 15-minute appointment. This study highlights the importance of this issue well.

An alternate interpretation

The analysis you see above is, unfortunately, all my own and not that of the study's authors. The study's authors, instead, believe in magic. Yep, the type of placebo given to one of the placebo groups was a homeopathic remedy. The study concludes that it is the shaken water that is the effective treatment, not the additional attention and care given by the treating doctor.

Indeed, the authors believe so strongly in magic that if a patient had an adverse reaction in either the drug or the other placebo group, they were shifted to the homeopathic remedy group. As a result of the authors' commitment, there is no consideration given to any possible alternative explanation. The statistical analysis says that these results are unlikely to be due to chance, so magic must be true.

You might ask how such a study got published. It shouldn't be that surprising, though—poor studies abound in the literature, and yet another poor study doesn't change much. PLOS One also has an ethos that makes this sort of paper more likely to get through. Its goal is to publish papers that are methodologically sound, irrespective of their results or significance—their reviewers are not asked to judge whether a study advances the field at all.

The idea is laudable because many papers are rejected on the basis of the perceived significance of their findings. Significance, of course, involves predicting the future, dooming reviewers to failure before they even start. I actually support PLOS One's idealistic approach. On that basis, a paper on homeopathy should be accepted if the methodology is sound.

In this case, though, I think PLOS One failed in its mission. The claim of the paper is that magic works. But the study was not rigorous enough to test that assertion. Instead, the researchers tested if additional personal care and attention from a medical professional (along with a placebo) worked. So it fails the sound methodology test.

Furthermore (and this is what really annoys me), we have had endless clinical studies on alternative medicine, each sucking up wads of cash. But because the vast majority of these alternatives rely on magic, there is never an investigation into their mechanisms. After all, how do you investigate the presence of meridian lines, energy flows, or the balance of humors if they don't exist? How do you even test for their existence?

Reviewing out of context

This is where the scientific publishing enterprise is letting us all down. During review, many reviewers focus solely on the paper and not the context. For example, if I were to publish a clinical study using some herb to treat a disease, there is the possibility that there is a mode of action that allows the herb to be effective (an herb has many thousands of chemicals, some of which could be therapeutic). So the clinical results can be judged in that context. Is there a dose response? Was the study well controlled? Has the data been analyzed appropriately?

In a case where there is a lot of evidence against any potential mechanism (or even the existence of said mechanism), then perhaps the reviewer should judge the study in that context. Who cares if the study was well controlled or if the data is analyzed appropriately—if there is no reasonable mechanism, shouldn't the paper you're reviewing throw some light on that fundamental question?

Or maybe the question is better phrased this way: are the results strong enough to make me doubt all of the fundamentals of physics, chemistry, and biology? In the case of this paper, the answer is emphatically no. The authors' favored mechanism has not been demonstrated in any way whatsoever. Until that changes, all further clinical studies should be rejected out of hand.

PLOS One, 2015, DOI: 10.1371/journal.pone.0118440