Authors: Marshall Schott & Greg Foster

With help from: Scott Janish & Justin Angevaare

Since starting the xBmt series, we’ve received critiques regarding the fact so many have returned non-significant results, many of which seem quite valid, such as the p-value being imperfect or our sample sizes too small. However, there’s one complaint we get more than any other, I can only imagine as a mostly benign knee-jerk response to data that doesn’t jibe with one’s convictions. I refer to it as the “shitty (bad) palates” argument and it entails presuming/accusing the xBmt participants of not being able to reliably distinguish a difference because they aren’t very good at tasting stuff.

Totally understandable, I’m guilty of making such assumptions, even though I know most of the people on my tasting panels and trust their palates more than my own. One of the wonderful aspects of science as opposed to other “systems” is that questioning of results isn’t only expected, it is encouraged! Hypotheses aren’t presumed to be correct based on hunches, but rather put under the proverbial microscope with the hope of gaining a deeper understanding of whatever it is being investigated. I’m reminded of something the great biochemist, Issac Asimov, once said:

The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” but, “That’s funny…”

Inspired by the search for something funny and curious whether our results were indeed impacted by our participants’ shitty palates, we compiled the data from over 30 of the most recent xBmts to see whether this argument holds any water. With the help of Scott Janish, we put together the following infograph to display the performance of participants based on their level of experience. Check this out.

Whaaa?! This took me by surprise, no doubt. I reached out to our statistician friend, Justin Angevaare, who blogs at On Brewing, to see if he might shed some light on the performance differences between BJCP judges and non-judges. Here’s what he had to say:

Logistic regression can be used to determine if there is a statistically significant link between successful performance on the triangle test and participant experience level. This method allows us to model a binary outcome on the basis of one or more explanatory variables, then the statistical significance of these explanatory variables can be tested. Here, the odds of being correct on the triangle test would be modeled by taster experience level.

Using this method, the odds of selecting the correct sample in the triangle test (the odd-beer-out) by participants holding a BJCP rank of provisional or higher were compared to the same odds for tasters who hold no BJCP rank. To make this comparison accurately, differences among individual xBmts had to be accounted for in the model, which was deemed necessary because the xBmts varied in terms of difference detectability and participant demographics.

In the end, there was insufficient evidence to suggest a difference between BJCP judges and those without such ranking in terms of odds of success on the triangle test (p<0.05; p=0.7698).

Interesting, eh? While it might be easy to assume a person who has invested the time and energy to become a BJCP judge has an extraordinary palate, these results appear to suggest they possess tasting abilities similar to, well… everybody else. Rather, what seems more plausible is BJCP judges are perhaps better at describing what they’re experiencing than others, much the way a sound technician is better than a veterinarian at talking about sound stuff. They know the language and what they should be looking for. It should be noted the triangle test is designed to examine only one’s ability to distinguish between samples based on flavor, aroma, and mouthfeel, participants aren’t asked to describe their experience, thus arguably placing judges and non-judges on a level playing field.

Are you saying there’s no value in becoming a BJCP judge?

Absolutely not! Many people find great meaning in the competitive and evaluative aspects of brewing. That’s awesome, I’d never suggest folks not do what it is that makes them happy. I’m a judge myself, and while I’ve always been highly skeptical of the between-judge reliability when it comes to scoring, it is a lot of fun and contributes to the building of a rad community, that’s what matters to me. More than anything, I think this data demonstrates the shitty palates argument is largely futile, which may come across as a bit self-affirming. Fine. I also think it serves as justification for what we’re doing, not that it increases the validity of the xBmt results in any way, but maybe, hopefully, it’ll reduce the worry some have that the non-significant results are more a function of the tasters’ inability to accurately detect differences because of lack of experience rather than the more likely scenario the beers just aren’t that different.

Finally, this is not conclusive proof of anything. Although the data seems to support the notion that those trained to identify and describe beer character are no better than general beer drinkers at distinguishing between the samples included in the 34 xBmts analyzed, it’s wholly possible this is a function of some factor we’re missing. And this is where opinion comes into play, as one’s beliefs about what it means to be a BJCP judge, and the convictions they hold about their own tasting abilities, is most certainly going to shade the manner in which this information is digested. And that’s okay, it’s to be expected, we’re only human.

Huge props to Greg for compiling and organizing the data, to Scott for helping put it all in a visually pleasing and easy-to-read format, and Justin for his willingness to lend his statistical genius. Cheers!

Support Brülosophy In Style!

All designs are available in various colors and sizes on Amazon!

Follow Brülosophy on:

| Read More |

18 Ideas to Help Simplify Your Brew Day

7 Considerations for Making Better Homebrew

List of completed exBEERiments

How-to: Harvest yeast from starters

How-to: Make a lager in less than a month

| Good Deals |

Brand New 5 gallon ball lock kegs discounted to $75 at Adventures in Homebrewing

ThermoWorks Super-Fast Pocket Thermometer On Sale for $19 – $10 discount

Sale and Clearance Items at MoreBeer.com

If you enjoy this stuff and feel compelled to support Brulosophy.com, please check out the Support Us page for details on how you can very easily do so. Thanks!

Advertisements

Share this: Facebook

Twitter

Pinterest

Tumblr

Email



Like this: Like Loading...