Ben Goldacre, The Guardian, Saturday 12 February 2011

One thing you hope for, with politicians, is that they won’t make the same mistakes over and over again.

Last week we saw that the government has overstated the problems in the NHS by using dodgy figures (to be precise, they used misleading static figures instead of time trends). We saw that Andrew Lansley’s repeated claim that his reforms are justified by evidence was untrue: the evidence doesn’t show that his price-based competition improves outcomes (if anything it makes things worse); and the evidence also doesn’t show that GP consortia improve outcomes (unless you cherry pick only the positive findings). It’s okay if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.

Now Lansley’s junior minister Paul Burstow MP has kindly responded, repeating the exact same mistakes again, only more clumsily. I find this, in all seriousness, genuinely frightening from a minister, so I’ll explain how he does it.

The government initially claimed that UK heart attack death rates were twice as bad as France. This was an overstatement: they are, but following recent interventions the gap is closing so rapidly that on current trends it will have disappeared entirely by 2012. In response, Burstow cites a 2008 paper by McKee and Nolte which he says “concluded that the UK had one of the worst rates of mortality amenable to healthcare among rich nations”.

Burstow either misunderstands or misrepresents this very simple and brief paper. It is a study explicitly looking at time trends, not static figures, and it once again finds that comparing 2003 with 1998, the UK still had fairly high rates of avoidable mortality, but these were falling faster than in all but one of the other 18 industrialised countries they examined (meanwhile in the US, avoidable mortality improved at a disastrously slow pace, although they spent more money).

This is a paper showing the success of the NHS, and the fact that we are discussing such a massive improvement in avoidable mortality from Labour’s first term in government is not my choosing: this is the paper that was cited by the Tory minister as evidence, bizarrely, of the NHS’s recent failures.

Next Burstow says I “overlooked the impact assessment we published alongside the health and social care bill, where we present a thorough analysis of the evidence for and against our plans… studies show that GP fundholding and practice-based commissioning delivered shorter waits and fewer referrals to hospitals for patients.”

In its section on GP fundholding, this “thorough analysis” ignores the four peer-reviewed academic papers I described last week, which sadly found no evidence of an overall benefit from GP fundholding. It makes a series of 5 assertions about outcomes, though these are unreferenced to any paper at all.

I contacted the Department of Health, who ferreted out the sources: there was just one, a document from the King’s Fund. It’s not a peer-reviewed academic journal article, but the King’s Fund are pretty good, in my view. If you read this document, it too finds that the results of GP Fundholding were mixed: some things got better, some things got worse.

So the Minister has cherry picked only the good findings, from only one report, while ignoring the peer-reviewed literature. Most crucially, he cherry-picks findings he likes whilst explicitly claiming that he is fairly citing the totality of the evidence from a thorough analysis. I can produce good evidence that I have a magical two-headed coin, if I simply disregard all the throws where it comes out tails.

Here is what politicians apparently cannot understand: it’s fine to make policy based on ideology, whim, faith, principles, and all the other things we’re used to. It’s also fine for evidence to be mixed. And it’s absolutely fine if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.