Recently I’ve run across a couple of New Zealand companies that sell therapeutic products – one a weight loss pill, the other a jet lag drink – that seem to put marketing first and let science take the back seat. This is by no means new behaviour, but I want to use them as examples to illustrate this widespread problem and suggest what can be done to combat it.

Before promoting a therapeutic product, you should first have good reason to believe that it works. This, I hope, is common sense, but it’s also enshrined in the Fair Trading Act and the Therapeutic Products Advertising Code as they prohibit unsubstantiated claims. This means you have to test the product, and do so rigorously. Rigorous clinical trials are expensive to undertake though, so they’re quite a prohibitive first step.

Instead of jumping straight into the deep end, a useful first step can be to undertake a smaller and less rigorous (and therefore less expensive) experiment. In order to answer the question of whether or not a product actually works you need to conduct a more rigorous trial. There’s a great cost involved in doing this, but if the results of a preliminary trial are optimistic then you have reason to expect a more rigorous trial might give similar results, so the expense might be worth it. You may even be able to get some funding to help with a more rigorous trial on the basis that the preliminary results were positive.

This is what Tuatara Natural Products has been doing with their weight loss pill “Satisfax”. They have completed a low quality preliminary trial on their product, which has been colloquially dubbed the “Fat Mates” trial. Although I don’t believe it has yet been published in a journal at the time I’m writing this article, it was registered retrospectively in the Australian New Zealand Clinical Trials Registry: Effect of the dietary supplement Satisfax (Registered Trademark) on weight loss in overweight volunteers. Some information on the data in the trial can be found on their website as part of an analysis by Dr Chris Frampton: Analysis of Satisfax® Clinical trial

As you can see on the trial registration page, this was an uncontrolled trial on overweight adults. The original plan was to recruit 100 volunteers with the hope that at least 60 will complete the trial. I have to say I’m a bit confused about how many people were in the trial, as apparently the recruitment was increased to 200 applicants after applications opened (a change that has not been reflected in the trial’s registration) yet apparently about 400 people applied. One article claims there were 200 participants, a later media release from Tuatara Natural Products seems to imply 100 were recruited, and the analysis of the trial says there were only 81 participants. Either way, 81 participants completed the full 8 weeks, and 52 of them took the recommended dose for the whole duration of the trial.

If there were 19 or 119 participants who didn’t complete the trial, the statistical analysis seems to ignore them with no justification given. This is unusual – a 19% drop out rate is significant and shouldn’t be swept under the rug. A lot of the time Tuatara also seems to ignore the 29 participants who didn’t drop out but also didn’t take the full dose for the whole duration.

The Science Media Centre posted the responses of 2 experts, Associate Professor Andrew Jull and Professor Thomas Lumley, to a press release from Tuatara Natural Products in February. It’s a good analysis of some of the weaknesses with the study, and I recommend you read it: Kiwi diet pill claims – experts respond

This trial was uncontrolled, and therefore also unblinded and unrandomised. As Professor Lumley explains, this is a problem if you want to draw strong conclusions from its results. It is of low methodological quality, but that’s okay. There is no problem with doing less rigorous trials first if they’re done in order to determine if more rigorous trials are necessary. Dr Glenn Vile, Chief Technical Officer of Tuatara Natural Products and the principal investigator for this study, wrote the following in a comment on a post on the “Fat Mates” trial by Dr John Pickering:

The Fat Mates trial was designed by clinical trial specialists to generate information about the Satisfax® capsules that would help Tuatara Natural Products plan a larger and longer double blind, cross over, placebo controlled trial. … We will use this information to proceed with the next clinical trial, but in the meantime we were so excited the weight loss achieved by most of our Fat Mates was much greater than the placebo effect seen in other weight loss clinical trials that we decided to launch the product so that anybody who is overweight can try Satisfax® for themselves. Dr Glenn Vile

I think the first part of what I’ve quoted above describes exactly what Tuatara Natural Products should be doing. They’ve conducted their low quality trial, and intend to use its results to proceed with a larger, longer, and more rigorous clinical trial. This is the right way to proceed – they now have an indication that their product might be effective, so they should do the research to find out.

The problem is that that’s not all they’re doing. After performing only a small low-quality trial, they’ve released their product for sale online and have been making a lot of noise about it. In my opinion, they’ve been significantly overstepping the results of their clinical trial. For example, in his comment Dr Vile also said:

our initial trial has shown [Satisfax] to be extremely effective in some overweight people. Dr Glenn Vile

In their media release on the 20th of February, they reported the average weight lost only by the 52 participants who took the full dose to completion (rounded up from 2.9 kg to “close to 3kg”) but not the average weight lost by all participants. They then reported in bold that the top 26 participants lost more weight, and the top two participants lost even more weight than that!

This cherry picking of the best results appears to have been part of Tuatara Natural Products’ marketing strategy for at least a few months now. In January, Stuff published an article on the trial highlighting the single person who lost the most while participating in it: Blenheim ‘fat mate’ loses 13.5kg in 8 weeks.

That article particularly highlights the person who lost the most weight out of all those in the trial, at 13.5 kg (confusingly, the maximum weight loss reported in the analysis of the trial’s results is 13.3 kg). However, she was one of only 2 participants who lost over 10 kg, and on average the 52 participants who took the recommended dose for the full eight weeks lost 2.9 kg. Losing 13.5 kg is very far from a representative example. I’m not surprised that they didn’t choose instead to focus on the participant who gained 1.2 kg despite taking the recommended dose for the whole duration, but that is actually much closer to the mean change in weight.

The article is, for all intents and purposes, one big testimonial in favour of Satisfax. It was an article, not an advertisement, which is important because in New Zealand it’s illegal to publish any medical advertisement that:

directly or by implication claims, indicates, or suggests that a medicine of the description, or a medical device of the kind, or the method of treatment, advertised… has beneficially affected the health of a particular person or class of persons, whether named or unnamed, and whether real or fictitious, referred to in the advertisement Medicines Act 1981 Section 58(1)(c)(iii)

This effectively bans all health testimonials from advertisements. I think this is a good part of the law, as testimonials can be both very convincing and completely misleading; a quack’s dream. Banning them should force businesses to instead focus on the results of research on their products, but this hasn’t stopped Tuatara Natural Products from getting stories written about the most extreme testimonials they could find from people who have lost weight at the same time as they were taking Satisfax.

More recently, Tuatara Natural Products has put out a press release multiple times (at least on the 20th of February and again on the 4th of March) that I think rather oversteps the results of their small preliminary trial:

A NEW ZEALAND SOLUTION TO A GLOBAL PROBLEM A little pill is providing an exciting answer to one of the worlds greatest and fastest growing problems: Obesity.A NEW ZEALAND SOLUTION TO A GLOBAL PROBLEM A little pill is providing an exciting answer to one of the world’s greatest and fastest growing problems: Obesity. Press Release – Tuatara Natural Products

I simply don’t think they are at all justified in saying that their new product is “providing an exciting answer to… Obesity”. They are putting marketing ahead of science, and that’s not okay.

Another company that seems to put marketing before research is 1Above. They make a drink which they claim can help you recover faster from jet lag, and have recently been in the news for signing a sponsorship deal with the fantastically successful golfer Lydia Ko.

At the end of that article about their sponsorship deal the reporter, Richard Meadows, made some comments regarding the science behind jet lag relief products and asked some good questions of 1Above’s CEO, Stephen Smith (emphasis mine):

[1Above’s] product contains a mixture of vitamins B and C, electrolytes, and Pycnogenol, a pine bark extract. The efficacy of flight drinks to combat the effects of jetlag is unproven. Late last year pharmacists were warned after the Advertising Standards Authority upheld a complaint against an ad saying a homeopathic anti-jet lag pill really worked. [1Above CEO Stephen] Smith said 1Above would not be doing clinical trials, which were highly expensive and not necessary. “What we tend to use is testimonials from people who have used the product and swear by it.” Smith said the key ingredient, Pycnogenol, had itself had been tested in dozens of trials, including its effects on reducing jetlag. Kiwi startup 1Above signs golf No 1 Lydia Ko

Yes, you read that correctly. The CEO of 1Above literally said that they won’t be doing clinical trials because they are “not necessary” and that they use testimonials instead.

As I said before, using testimonials to promote a therapeutic product, like a drink to help you recover faster from jet lag, can be both very convincing and completely misleading. There’s a reason why testimonials implying health benefits are illegal in New Zealand, and I hope that 1Above’s marketing will not violate this regulation.

Not all testimonials are prohibited, of course. It’s entirely acceptably to provide a testimonial from someone who thinks their drink tastes great, or that they provide great service. Basically anything for which a single person’s experience can provide a useful insight. Therapeutic effects, almost without exception, do not fall into this category, which is a big part of why we need to do clinical trials in the first place. If they quote someone in saying that their product helped them recover faster from jet lag, they may be in danger of breaching the Medicines Act.

For example, I’d expect they probably shouldn’t use a testimonial that says this:

On their website, 1Above currently does refer to research on one of the ingredients in their product, “pycnogenol”. Professor Lumley recently wrote a post about this on his other blog, Biased and Inefficient, regarding these studies and how they are used by 1Above: Clinically Proven Ingredients

I recently contacted 1Above to ask about some discrepancies I found between the abstract of the study they cited for showing pycnogenol reduced the duration of jet lag and their description of it on their website:

I was interested to see the claim your company made that Pycnogenol® has been shown to support circulation and reduce the length and severity of jet lag. I have found the study “Jet-lag: prevention with Pycnogenol. Preliminary report: evaluation in healthy individuals and in hypertensive patients” that is mentioned on your website as the source for this claim, but I am only able to access the abstract of this preliminary report. Unfortunately, as far as I can tell, the study protocol didn’t involve blinding of participants or researchers. The participants in the study took 50 mg Pycnogenol 3 times per day, but I haven’t been able to find out how much is contained in your products. Is this information available anywhere on your website? I notice the study also says the participants took this regimen for 7 days, starting 2 days prior to departure. Is this comparable with how your product is intended to be used? I also noticed some differences between the description of the study and its results between the abstract and your website, I would be grateful if you could explain to me the source of these differences. The abstract states the control group took, on average, 39.3 hours to recover and the experimental group took, on average, 18.2 hours to recover. However your website reports these as 40 and 17 hours respectively. Also, your website states that the study involved 133 passengers (it’s not clear from the description on your website if they all took Pycnogenol or if some of them were in the control group) who reported the time it took them to recover from jetlag. However, the study’s abstract states that in the first experiment, which is the only one that involved the reporting of the time taken to recover from jetlag, only involved 68 participants – 30 in the control group and 38 in the experimental group. I would be grateful if you could explain these differences to me, and if you could send me any other relevant scientific information that supports this claim.

To their credit, since receiving my message they did update their website to fix the discrepancies in the reported number of participants and times taken to recover from jet lag, and their CEO replied to thank me for pointing these discrepancies out.

However, they didn’t respond to my other questions about the amount of pycnogenol in their products or the study involving the participants taking pycnogenol for 7 consecutive days, starting 2 days before their flight, which is inconsistent with how 1Above recommends their products be used.

This is just one more company basing their marketing on preliminary trials instead of using them as the basis for research that could actually answer the question of whether or not a product is useful. Worse than Tuatara Natural Products, they even go so far as to consider clinical trials “not necessary” and apparently intend to rely on testimonials instead. It would be much more appropriate for them to spend some of their $2.4 million annualised income on researching their product rather than paying for a sporting celebrity to endorse them.

I try to make my rants constructive, so I want to end this article with the question “What can we do about this?”. If you have any suggestions, I’d love to hear them in the comments section.

I think the most important thing that anyone can do to address this problem is to ask for evidence. If you see a claim made about a product that you think you might buy, then get in touch with the company selling it to let them know you’re considering buying it and to ask for evidence. If they don’t have a good enough answer, then let them know that’s why you won’t be buying their product. If they give you evidence to back up their claim, then great!

The UK organisation Sense about Science has created a website for just this purpose: www.askforevidence.org

Asking for evidence doesn’t have to be a big deal, involving a formal letter or anything like that. When you see a weight loss product advertised on a one day deal site, a copper bracelet that apparently offers pain relief advertised on a store counter, or a jet lag cure promoted on Twitter, make your first response be to politely ask for evidence.

This isn’t a problem that’s going away any time soon. As consumers, we deserve to be able to make informed decisions about the products we buy, and when companies put marketing before research it becomes harder to make these informed choices. But if we work together then we can encourage companies like Tuatara Natural Products and 1Above to improve their behaviour and attitudes toward marketing and research.

Let’s turn “what’s the evidence?” into a frequently asked question for all companies that sell therapeutic products.