Ben Goldacre, The Guardian, Saturday 1 August 2009

This week the Food Standards Agency published 2 review papers showing that organic food is no better than normal food, in terms of composition, or health benefits. The Soil Association’s response has been swift, receiving prominent and blanket right of reply: this is testament to the lobbying power of this £2bn industry, and the cultural values of people who work in the media. I don’t care about organic food. I am interested in bad arguments. Theirs has three components.

Firstly, they say that the important issue with organic food is not personal health benefits, but rather benefit to the environment. This is a popular strategy from losing positions: “don’t talk about that, talk about this“. It is diversionary, and we can separately talk about the environmental issues with organic food, but right now, we are talking about the health benefits.

Secondly, they say that the health benefits of organic food are related to pesticides, are positive, and cannot be measured by the evidence that has been identified and summarised in the FSA paper. This, again, is gamesmanship.

Either you are proposing that there are health benefits which cannot ever be measured (because it is too hard to disentangle the health benefits of eating organic food from other beneficial features of peoples’ lives, for example, when you measure their health in a 30 year study). In this case you have faith, which is not a matter of evidence.

Or you are proposing that there are health benefits which could be measured, but have not been yet. In this case, again, you have faith rather than evidence, but you could at least start recruiting researchers now, using your £2bn, to investigate your beliefs with fair tests.

And thirdly, sadly, like many industries in a corner, the Soil Association seek to undermine the public’s understanding of what a “systematic review” is (which itself causes collateral damage to everybody’s ability to engage in debates on evidence).

They say that the report has deliberately excluded evidence to produce the answer that organic food is no better. The accusation is one of “cherry-picking”, and it is hard to see how it can be valid in the kind of study conducted by the FSA, because in a “systematic review”, before you begin collecting papers, you specify how you will search for evidence, what databases you will use, what types of studies you will use, how you will grade the quality of the evidence (to see if it was a “fair test”), and so on.

Now: what are the Soil Association angry about the FSA report ignoring? As an example, from their press release, they are “disappointed that the FSA failed to include the results of a major European Union-funded study involving 31 research and university institutes and the publication, so far, of more than 100 scientific papers, at a cost of 18million Euros, which ended in April this year”. They gave the link to www.qlif.org.

I followed this link and found the QLIF list of 120 papers. Almost all are irrelevant. The first 14 are on “consumer expectations and attitudes”, which are correctly not included in a systematic review of the evidence on food composition. Then there are 22 on “effects of production methods”: here you might expect to find more relevant research, but no.

The first paper (“The effect of medium term feeding with organic, low input and conventional diet on selected immune parameters in rat“) while interesting, will plainly not be relevant to a systematic review on nutrient content. The same is true of the next paper, “Salmonella Infection Level in Danish Indoor and Outdoor Pig Production Systems measured by Antbodies in Meat Juice and Faecal Shedding on-farm and at Slaughter“: it is not relevant.

Furthermore, the overwhelming majority of these are unpublished conference papers, and some of them are just a description of the fact that somebody made an oral presentation at a meeting. The systematic review correctly only looked at good quality data published in peer reviewed academic journals, and with good reason: we know that conference papers are unreliable sources of information, often change between conference and publication, and, in fact, we know that conference papers are often never published at all.

This raises the issue of transparency: we want the methods and results of scientific research to be formally presented, and accessible by all. If a government report on anything relies substantially on unpublished and inaccessible research then we are correctly concerned: in fact, I raised such concerns, two weeks ago, because the key piece of evidence presented by the Home Office to justify retaining DNA from innocent people who have been arrested was an incompetently presented piece of unpublished and incomplete research.

I could go on, but in reality, this is not about organic food. The emotive commentary in favour of organic farming bundles together diverse and legitimate concerns about unchecked capitalism in our food supply: battery farming, corruptible regulators, or reckless destruction of the environment, where the producer’s costs do not reflect the true full costs of their activities to society, to name just a few. Each of these problems deserve individual attention.

But just as we do not solve the problems of deceitfulness in the pharmaceutical industry by buying homeopathic sugar pills, so we may not resolve the undoubted problems of unchecked capitalism in industrial food production by giving money to the £2bn industry represented by the Soil Association.