One of the frustrating aspects of how science is reported in the mainstream media is when a complex study with very unclear results is presented with a misleading bottom line. Most people read only the headline, or perhaps the first paragraph, in order to glean the essence of a scientific study. They don’t read deep into the reporting to find the important details, or go to the study itself.

This is especially problematic when the study is of a preliminary design, or when the author’s conclusions are biased or misleading.

The most recent example of these issues is a study looking at the consumption of organic food and the risk of cancer. CNN reported the study as showing: “You can cut your cancer risk by eating organic, a new study says.” No – that is not what the study shows.

The study itself is not bad, for what it is, but it is highly limited in the conclusions that can be drawn from it, and it has some serious limitations. The researchers looked at a French database of good consumption, NutriNet-Santé Prospective Cohort Study. They had volunteers fill out food diaries for three days, report their organic food consumption, gathered demographic and other lifestyle data, and then followed them for over four years, using various methods to track the incidence of cancer.

They found a negative association between organic food consumption and cancer risk only for the highest organic food consumption group, and only for those with an overall low to medium quality diet, and only for two types of cancer (postmenopausal breast cancer and non-Hodkin lymphoma). These are hardly consistent or compelling results. While they tried to account for confounding variables, the complexity of this study essentially makes that impossible. The results are therefore impossible to interpret with any confidence.

To clarify – this is an observational study only. This means we cannot make any firm cause and effect conclusions, only look for correlations. Further, the correlations are extremely complicated. Here are the problems which I think make the results uninterpretable.

Here is the strangest outcome:

Combining both a high-quality diet and a high frequency of organic food consumption did not seem to be associated with a reduced risk of overall cancer compared with a low-quality diet and a low frequency of organic food consumption. Negative associations were found between the risk of cancer and combining both a low- to medium-quality diet and a high frequency of organic food consumption (eTable 6 in the Supplement).

It is hard to know what this means. At face value it seems that if you have a good diet, switching to organic adds nothing. If you have a low to medium quality diet, only then was eating organic food associated with some lower cancer risk. But the potential confounders here are massive.

Further, the population in this study was not random – they were volunteers. They were 3/4 women, who are more likely to eat organic in the first place. They report:

Higher organic food scores were positively associated with female sex, high occupational status or monthly income per household unit, postsecondary graduate educational level, physical activity, and former smoking status (Table 1).

They tried to account for these variables, but that is really not possible in this kind of study. Clearly the people in this study, who were volunteers and not randomly selected, behave differently than the general population. You might capture some of their behaviors, but not all.

The one methodological quibble I have is that there is no mention of controlling for multiple variables and comparisons. When you look at multiple outcomes and multiple potential associations, having one or two be significant is not surprising.

Here is the fatal flaw, which they acknowledge:

“Some limitations of our study should be noted. First, our analyses were based on volunteers who were likely particularly health-conscious individuals, thus limiting the generalizability of our findings. NutriNet-Santé participants are more often female, are well educated, and exhibit healthier behaviors compared with the French general population. These factors may may have led to a lower cancer incidence herein than the national estimates, as well as higher levels of organic food consumption in our sample.”

They also acknowledge that their results are not consistent with the far larger Million Women Study, which included 623,080 UK women and concluded:

In this large prospective study there was little or no decrease in the incidence of cancer associated with consumption of organic food, except possibly for non-Hodgkin lymphoma.

This larger study was not mentioned in CNN’s reporting.

This study also represents the main problem I have with organic farming – it is a false dichotomy based on the appeal to nature fallacy. The authors conclude that their study suggests promoting consumption of organic food may be a way to reduce cancer. Their study is not capable of showing this, but even if there is some real signal in this messy data, that is the wrong approach.

Promoting organic food is not sustainable. We simply don’t have the land, or the manure, to feed any significant portion of the world with organic food. Further, the organic lobby takes many unscientific stands, such as opposing food preservation through irradiation and all genetic modification. It is a package deal, with lots of nonsense and pseudoscience.

What we need is to take an objective evidence-based look at each individual variable. It is certainly possible that some pesticide is causing adverse health effects at the currently used levels (although there is no current evidence for this). If so, we need to find out which one, and either eliminate it or reduce the safety threshold. This is in fact what the various regulatory agencies do – monitor the scientific data and adjust safety thresholds accordingly, with a generous buffer.

There are also a host of negative unintended consequences possible with promoting organic farming. Organic produce is usually significantly more expensive, and may reduce overall fruit and vegetable consumption if people are avoiding more affordable produce out of unwarranted fears.

Let’s also keep in mind that many pesticides are allowed in organic farming, they just have to be “natural”.

The very concept of organic farming is a pseudoscientific nuisance. This study is just one example of the confusion it creates.