Are organic wines better than conventional ones? An interesting study from researchers at the University of California Los Angeles suggests that they might well be. However, the difference in point scores between eco-certified wines and conventional wines is actually much smaller than press reports on this study have stated, because they haven’t read the paper carefully.

This study examined critic ratings of almost 75 000 Californian wines, with vintages ranging from 1998 to 2009, from just over 3800 wineries. The scores were taken from The Wine Advocate, Wine Enthusiast, and Wine Spectator. The researchers then checked to see whether any of these wines were certified as organic or biodynamic from those wineries (in those vintages), to create a subset of what they call eco-certified wines which they then compared with conventional wines.

[As an aside, the analysis of scores is really interesting. The authors published the distribution of scores from each of the wine publications, and also together, and found that they roughly followed a normal distribution with the median score in this distribution being 89. But they found a rounding up effect, in that many of the 89 point scores appeared to have been rounded up to 90. There are fewer wines scored at 89 points (5153 wines) than there are at 88 (7584 wines) and at 90 (6989 wines).]

Only 1.1% of the wines in the sample were eco-certified, but being eco-certified increases the score of the wine by 4.1 points. But the important detail here is that we are not talking 4.1 points on the 100 point scale, and this is something that no one who has written about this has emphasized or even picked up, as far as I know. This is 4.1 points on a standardized scale devised by the authors, which has a much broader discriminatory power, because it is more spread out than the very bunched 100 point scale. This detail really does change everything. The standard deviation of the 100 point system is around 4 for these three publications, whereas it is 28 for the scaled score, a sevenfold difference. So the advantage of being organic on the 100 point scale is much smaller than 4.1 points!

It also needs to be clarified that this result doesn’t mean that converting to organics or biodynamics is going to raise the score of your wine 4.1 points on this standardized point scale. There is no evidence for causation here.

In the first instance, there is no stratification by price in this study. It could be that the average price of the eco-certified wines could be a lot higher than that of the average of the conventional wines, and as there would likely be a strong correlation between price and wine score this would be a confounder. Also, it could be a selected subset of higher-achieving wineries that makes the switch to organics. It takes an ambitious, conscientious producer to switch from conventional farming to organics. In California, organics is a very small subset of all wines.

The other problems with this study is that quality is measured though critic ratings. If you look at the particular critics who gave those ratings, then the question is one of whether you think their palates actually differentiate quality in a meaningful way (to your own particular palate).

Despite any evidence for a causal link, there remains the possibility that farming organically could improve wine quality. Potential mechanisms exist, and I think that there is a lot of potential benefit to farming vines with healthy, living soils (although certified organic/biodynamic is not the only way to achieve this). But this study shouldn’t be taken to show that shifting to organics results in better quality wines, because it doesn’t.