Are organic foods better for you? A study published in the British Journal of Nutrition appears to indicate so and some media outlets that have covered the study seem to agree uncritically, but independent scientists are raising red flags about the claims.

The study, “Higher antioxidant and lower cadmium concentrations and lower incidence of pesticide residues in organically grown crops: a systematic literature review and meta-analyses,” is a meta-analysis of 343 previously-published studies that compared the nutritional composition of organic versus conventional foods. The team of scientists led by Carlo Leifert, professor of ecological agriculture at Newcastle University in England, did not conduct any new original laboratory or field work. The study was funded by the European Union and the Sheepdrove Trust, a British charity that funds research in support of organic farming–in other words, it’s in part an organic industry funded study.

The Los Angeles Times covered the study with a sweeping headline, “Organic foods are more nutritious, according to review of 343 studies.” “Research is first to find wide-ranging differences between organic and conventional fruits, vegetables and cereals,” said an article in the Guardian that broke the embargo on the research last week. Leifert and his team found “substantially” higher levels of antioxidants and lower levels of pesticides in organic fruits, vegetables and grains than in conventional produce, according to the New York Times.

“It shows very clearly how you grow your food has an impact,” Leifert told the Times. “If you buy organic fruits and vegetables, you can be sure you have, on average, a higher amount of antioxidants at the same calorie level.”

But many in the scientific community are skeptical of the methodology and critical of the broad conclusions drawn in the study, including several researchers quoted in the Guardian and the New York Times. Tom Sanders, professor of nutrition at King’s College London, told the Guardian that the study has been “sexed up.”

There have been three other similar meta-analyses since 2009, which have all concluded that there are few, if any, differences in the nutritional content of organic and conventionally grown foods. The first, a review of 137 studies by scientists from the London School of Hygiene and Tropical Medicine published in the American Journal of Clinical Nutrition in 2009, concluded:

On the basis of a systematic review of studies of satisfactory quality, there is no evidence of a difference in nutrient quality between organically and conventionally produced foodstuffs.

Another meta-analysis reviewed 237 studies comparing organic and conventional foods. This study, conducted by a team of Stanford scientists, concluded:

The published literature lacks strong evidence that organic foods are significantly more nutritious than conventional foods.

The authors of the new study, including Washington State University professor and long-time anti-GMO activist Charles Benbrook, say that the new study is more robust than the previous ones, partially because of the fact that more studies were included in the review.

But Alan Dangour, food and nutrition scientist at the London School of Hygiene and Tropical Medicine and lead author of the first meta-analysis study, disagrees, saying that weaker studies should have been excluded from the analysis.

“The quality of the available data varies greatly and it is therefore very surprising that, in their analysis, the authors decided to include all the data that they found, irrespective of their quality,” he said. “Mixing good quality data with bad quality data in this way is highly problematic and significantly weakens confidence in the findings of the current analysis.”

Hank Campbell, science writer at Science 2.0, explains how such meta-analyses can be massaged to provide desired outcomes when bias is not eliminated:

In a review, they look at no data, of course, and 343 papers becomes the problem rather than the solution when the methodology is flawed. Meta-analysis, as everyone with statistics knowledge knows, can boost the strength of systematic reviews when done properly but easily suffers from bias unless the researchers are truly interested in controlling eligibility criteria and methodological quality. Without controlled eligibility, it’s easy to find any pattern you want.

Some have pointed out that the new study is not independently funded, as opposed to the previous three other studies. Marion Nestle, professor of public health, nutrition and food studies at New York University cited in the New York Times on this study, writes in her blog:

One of the funders is identified as the Sheepdrove Trust, which funds research in support of organic and sustainable farming. … The paper says “the Trust had no influence on the design and management of the research project and the preparation of publications from the project,” but that’s exactly studies funded by Coca-Cola say. It’s an amazing coincidence how the results of sponsored studies almost invariably favor the sponsor’s interests. And that’s true of results I like just as it is of results that I don’t like.

Others note that the study targets ten groups of chemicals, including conventional pesticide residues, antioxidants and metals, but does not examine other pertinent chemicals such as pesticides commonly used in organic agriculture that are approved by the National Organic Program. While synthetic pesticides are not used in organic agriculture, organic pesticides such as rotenone and pyrethrin are used as these chemicals are produced by plant sources and considered ‘natural,’ even though they might be more toxic to people than some synthetic alternatives.

“None of the reviewed studies measured any remnants of so-called ‘organic’ pesticides–that is those that are permitted under the National Organic Program,” said Ruth Kava, senior nutrition fellow from the American Council on Science and Health. “There are many of these, and they are not necessarily safe for human consumption, but the organic foods industry seems to want to keep them secret.”

Additionally, there has been little to no reference to the undesirable results found in the study. “The paper also reports a decrease in protein, nitrates and fibre in the organically grown crops, which may be undesirable,” Richard Mithen, leader of the food and health program at the Institute of Food Research, noted, “and which are maybe unsurprisingly not referred to by the authors in their advocacy of organically grown produce.”

“The public health significance of the reported findings have been worryingly overstated,” Dangour added, saying that there has been no good evidence that suggests more antioxidants would have important public health benefits.

The nutritional composition of fruits and vegetables is affected by a large number of factors, from the soil nutrient levels to the time of harvest to how the produce has been handled and transported, according to Bruce Chassy, professor emeritus of food science and human nutrition at the University of Illinois at Urbana-Champaign. With so much variation and confluence of factors, it can be very misleading to say that any method of agriculture produces more nutritious foods than another based on research examining compositional differences in produce.

“The research (as is true for all crop research) is a photo in time of a single crop in a single field in a single growing season. There is an abundance of evidence that the next year or the next field will yield different results,” said Chassy. “There are nutrient differences between peas in the same pod, and tomatoes picked at 4 different times during the day. One should therefore take any composition research with a large grain of salt.”

There may be many disagreements over whether organic or conventional produce is more nutritious within the scientific community, but one thing is clear: simply encouraging more people to eat more fruits and vegetables, organic or not, will have more far-reaching effects on human health.

“The additional cost of organic vegetables to the consumer and the likely reduced consumption would easily offset any marginal increase in nutritional properties, even if they did occur, which I doubt,” Mithen said. “To improve public health we need to encourage people to eat more fruit and vegetables, regardless of how they are produced.”

Additional Resources: