Ben Goldacre, The Guardian, Saturday 27 February, 2010

After the Science and Technology committee report this week, and the jaw dropping stupidity of “we bring you both sides” in the media coverage afterwards, you are bored of homeopathy. So am I, but it gives a very simple window into the wider disasters in all of medicine.

Homeopathy, our first example, is a small sector of the pharmaceutical industry, a few sugar pill companies worth a couple of billion pounds a year in europe. Overall, trials show their pills perform no better than placebo. We therefore know that all claims to the contrary are bullshit, but bullshit being tolerated by plenty of MPs, huge swathes of the media, a fair few GPs, and most worryingly of all, the Medicines and Healthcare Regulatory Authority, who permit homeopathy pill companies to list diseases they say they can treat on the side of the bottle, with no requirement that they come up with any evidence that their treatment works.

This doesn’t only tell you a story about homeopathic sugar pills: this tells you how doctors, politicians, the media, and regulators deal with the issue of evidence in medicine.

How closely do the great and the good, for example, scrutinise the promotional material for medical drugs? The latest paper looking at this question is published this month. Researchers in Holland went through all the biggest medical journals in the world – the Journal of the American Medical Association, the Lancet, the New England Journal of Medicine, and so on – between 2003 and 2005. All adverts were included, once each, if they made a claim about the effect of a drug. For all the claims in the ads, they checked the references, found the trials they referred to, and then gave them out to an easily exploited workforce of assessors: 250 medical students who’d just finished their evidence based medicine teaching.

Each student independently assessed 2 trials, and its associated advert, using a pre-specified questionnaire and a well-established scoring system to assess the quality of trials. Scores were given for things like: whether the method of randomly assigning patients to one treatment or another was adequate, and clearly described; whether patients could know which treatment they were getting; whether drop-outs were appropriately included in the analysis, and so on. These are good measures of whether a trial is a “fair test” of a treatment.

By now you will rightly be worrying that medical students – although cheap and easy to come by – are not reliable raters, so you will be pleased to hear that each trial was scored by between 2 and 6 students, and if there was a discrepancy in scores, that trial was reviewed by a panel of 4 academics.

The results were abysmal. Only half of the claims in the adverts were actually supported by the specific trials they referenced, and of all the trials, only 55% got a score of “high quality”: overall, only 39.2% of these adverts – in the leading medical journals in the world – referenced a high-quality trial which actually supported their claim.

This is not the first time such a study has been conducted. Villanueva and colleagues, in 2003, published a paper in the Lancet assessing claims for cardiac medication adverts in six Spanish medical journals: of the 102 references they could trace, 44% did not support the promotional statement. Similar results have been found in psychiatric drug adverts, and in the field of rheumatology. To offset any suggestion that I am cherry-picking, a systematic review of this question in the open access journal PLoS ONE found 24 similar studies, and overall only 67% of the claims in adverts were supported by a systematic review, a meta-analysis or a randomised control trial.

Quacks see the shortcomings in medicine as a justification for their own even more dubious behaviour, but in reality, the horror is this: homeopathy is the obvious, easy, tip of the iceberg. It is the simplest and clearest story of how bad things are, how dumb doctors and politicians can be about evidence, and how lame regulation has become. But it is only the most obvious illustration of the dark, fearsome depths into which these problems extend. We are in very big trouble.