A recent systematic review and meta-analysis of the effects of mindfulness meditation on prosocial behavior found, essentially, that there is no evidence that it works. I find these results entirely unsurprising, and they yet again highlight the need for rigorous research before concluding that a phenomenon is real.

As I discussed recently on SBM, mindfulness meditation is the practice of sitting quietly, focusing inward and on the present, and avoiding mind wandering or daydreaming. The recent review I discussed on SBM found that the research into mindfulness, however, does not use a uniform or operationalized definition. That is critical to good science – you need to carefully define something before you can do research on it.

It is especially important to specifically define a concept in order to do research into the question of whether or not the phenomenon is real. If your question is, “Does X exist,” you better have a very specific definition of what X is. Otherwise it is easy to misinterpret the evidence, or to wiggle out of evidence that X does not exist.

The best example of this in medicine is acupuncture. Acupuncture is defined as sticking thin needles into acupuncture points – except when research shows that it does not matter where or even if you stick the needles, then acupuncture can be something else, which is vaguely defined.

Once you have a specific definition, with clearly identified variables, then you can study if those variables which constitute the phenomenon in question have a specific effect. For mindfulness – what is the effect of relaxation, introspection, or avoiding daydreaming? Does mindfulness differ from other methods of achieving these same effects? Perhaps mindfulness is nothing more than relaxation, or perhaps any perceived effect comes from being distracted for a time from the stresses of your life.

In other words, mindfulness may not be a real distinct thing, but just one method of achieving other more fundamental states, such as relaxation. This is what appears to be the case, based on years of research, in which case proponents should claim, “Mindfulness is an effective method of relaxation, which can have benefits,” not “Mindfulness is a unique phenomenon with specific and unique benefits.”

Does this matter? Absolutely. It is almost guaranteed that when I post an article such as this someone will say, “Who cares, as long as it works.” But this is an unscientific attitude. In science, the details do matter. We need to know what is really real, because it affects how we implement interventions and how further scientific research proceeds.

Using acupuncture as an example again – if the sticking of the needles adds no specific effect or value, and all the benefit derives from the interaction with the acupuncturists (which is what the research clearly shows), then we can dispense with the needles. The needles are invasive and come with risk. Further, we don’t need to speculate about the mechanism of benefit from sticking people with needles, because there is no mechanism. We can shift our focus to the real phenomenon – a subjective effect from a positive therapeutic interaction.

With mindfulness, because there is nothing invasive, the situation is the same, if less obvious. If, as the recent review shows, watching nature documentaries is as effective as mindfulness, then you can simply turn on the TV, learn something about nature, and get all the apparent benefit of meditation. You also don’t have to spend hundreds of dollars on that mindfulness course. We can further stop wasting our time researching a scientific dead end.

Let’s get back to the recent review to see those details. What the researchers found when they reviewed the literature on prosocial behavior is that the research did not establish that mindfulness had any specific benefit. First, we need to define prosocial behavior more specifically. The authors write:

Five types of social behaviours were identified: compassion, empathy, aggression, connectedness and prejudice. Although we found a moderate increase in prosociality following meditation, further analysis indicated that this effect was qualified by two factors: type of prosociality and methodological quality. Meditation interventions had an effect on compassion and empathy, but not on aggression, connectedness or prejudice.

That’s important to know for research, but not really the most interesting finding of the research. They also found:

We further found that compassion levels only increased under two conditions: when the teacher in the meditation intervention was a co-author in the published study; and when the study employed a passive (waiting list) control group but not an active one.

So studies were only moderately positive when one of the study authors were teaching the subjects meditation. This suggests that researcher bias is at work.

Perhaps more importantly, however, is the fact that there was only a measured effect when there was a waiting list control, meaning there is an unblinded comparison where the subjects had no intervention. When the control was “active”, meaning the control group had some intervention, then there was no effect. This intervention could simply be watching a nature documentary. This one fact alone means the research is negative. Everything else is interesting, but doesn’t really matter.

The meta-lesson here is that this kind of analysis of the entire literature on a specific scientific question is necessary before you can come to any reliable conclusion about efficacy. If you want to know if an alleged phenomenon is real, you need rigorous research in which variables are clearly defined and adequately controlled for. Further, you need positive results with a clinically significant (adequate signal to noise ratio), statistically significant, and independently replicated effect. Until you get to that threshold, you are likely just dealing with researcher bias, p-hacking, publication bias, and loose methodology creating the illusion of a positive effect.

With some alleged phenomena, however, we never get to the threshold of acceptance. The research just goes around in circles chasing its tail. This is true of acupuncture research, mindfulness meditation, ESP, homeopathy, and essentially all the familiar pseudosciences. All we get are excuses, hyping of preliminary research, cherry picking positive studies, and personal attacks against skeptics who would dare to question the alleged phenomenon. In medicine we also get the, “Who cares, as long as it works.” This, of course, misses the point that the alleged treatment doesn’t really work, it is all an illusion.

Another defensive response is to claim, “Well, that is not the real claims being made for X. It’s really about this other thing over here.” This kind of response, however, is usually just part of a dance of avoidance. “They didn’t study real astrology.” “That is not the real reason to fear GMOs.”

But this approach, which often is just motivated reasoning, further misses the point that the person making the claim has the burden of proof. It’s not on me to prove with high quality research that mindfulness doesn’t work for every possible claim made for it. Proponents have to adequately demonstrate that it does work for a specific claim, and they haven’t. Scientists will then conclude, “The research does not justify rejecting the null hypothesis.” This is technically true, as is appropriate in scientific discourse. When communicating to the public, however, it makes it seem like we don’t really know the answer.

At some point (and where this point is admittedly requires judgement, which includes an evaluation of plausibility) a lack of evidence can be treated as, “OK, this probably doesn’t work.” At the very least, we can conclude that this is a scientific dead end.

There is also an asymmetry in the media. A systematic review like this, concluding that the evidence is inadequate to support a conclusion that mindfulness is effective in promoting prosocial behavior, just doesn’t get that much play. However, a crappy preliminary study with poor methodology and brimming with bias and p-hacking, which shows an apparent tiny effect, will be promoted in the media as proving that mindfulness works magic. Rarely will such studies be put into the context of the entire literature.

Media and marketing forces tend to lead to an adoption of new ideas long before they are adequately demonstrated. Then once they are embedded in the culture and the popular consciousness, they are hard to eradicate. The public ends up believing a lot of stuff that is simply not true. Then when skeptics or scientists point out that the research was never adequate to conclude the phenomenon is real, and now after 20 years or so we can more clearly say it probably isn’t, they seem out of touch. Everyone already knows that antioxidants are great for you, never mind that the research shows they have no benefit.

We can now probably add mindfulness to the list. I was never impressed with the research, and the claims always seemed poorly defined. Now, we have multiple systematic reviews which show essentially that. We may not be at the point where the concept can be completely abandoned, but we are at least getting close. The most parsimonious interpretation of the science at this point is that mindfulness is just a ritualized form of relaxation, with no specific benefit beyond that. If you enjoy it and find it useful, fine. If you prefer watching Blue Earth II, then I’m with you. Just don’t spend hundreds of dollars on courses, tapes, or seminars because you bought the hype.