Evidence can modify our beliefs, but the impact it has depends upon those beliefs. An 18th century priest has something to say about that, in what could be seen as a mathematical formulation of the scientific method

In my last post I dipped my toe into some statistics, to try to explain why the (essentially arbitrary) “five sigma” criterion for a discovery is so widely used in particle physics. As part of the explanation I quoted Louis Lyons saying that the 5 sigma requirement includes a “subconscious Bayes factor”.

Statistics gets some people excited¹, especially, it seems, when Bayes is mentioned. There was even a rapid response blog from (Bayesian) cosmologist at Sussex, Peter Coles.



One of the things that gets people fired up is that Bayesian statistics can introduce a level of subjectivity into the scientific process that some scientists see as unacceptable; whereas its counterpoint, the frequentist approach, can be seen (as Peter puts it) as answering the wrong question and is certainly prone to highly subjective (mis)interpretations. Emotive stuff. At this point I probably ought to make my own attempt at saying what Bayes’ theorem actually is, so here goes.



On one level it is an uncontroversial statement about conditional probabilities. Here people tend to start talking about “event A” and “event B” and stuff, which is ok, but a bit abstract. I find it easier to be concrete. So imagine I have a bag containing three stones; two blue and one red. Without looking, and in random order, you and I pick, and keep, one stone each. What are the chances I have blue and you have red?

I could work them out two ways. If you have the only red stone (which you have a one-in-three chance of having got, without knowing anything about my choice) then I must have a blue (one-in-one). The probability is ⅓ × 1 = ⅓, a third.



On the other hand, if we know I have a blue stone (probability two-in-three) then there is a 50:50 chance you have a red stone. The probability is ⅔ × ½ = ⅓ again.

The answers had to come out the same, since both ways of working it out describe the same result. The “probability of me having blue if you have red, multiplied by the probability of you having red”, has to be the same as “the probability of you having red if I have blue multiplied by the probability of me having blue”. Abstracted, that’s Bayes’ Theorem.



To see how it can be useful, imagine we only know that I have a blue stone. I can rearrange Bayes’ theorem to work out the chance you have a red stone. This “probability of you having red if I have blue” is equal to “probability of me having blue if you have red, multiplied by the probability of you having red, divided by the probability of me having blue”. All I did was divide both sides by the probability of me having blue. If we know those initial probabilities, this is all trivial.

And none of that is controversial. To a statistician it is basic stuff, used in many applications. Where debate gets more heated is the common, real-life situation where we don’t have complete information, and we are trying to work out what might be going on.



Imagine that we don’t know what colours the stones in the bag are (blue or red), and we don’t know what colour stone you have; but we do still know that I have a blue stone. What’s the probability that you have a red stone? As we don’t know how many of the three stones are blue and how many are red, we don’t know the “prior probabilities” – that is, the independent “probability of me having blue” and “probability of you having red”.

Bayesian inference would put those prior probabilities in terms of belief – the “epistemological interpretation”. What do I believe about the initial contents of the bag of stones? Do I have any prior evidence as to what the contents might be? If not, I would probably be best saying there’s an equal chance of getting either colour. That is, the prior “probability of me having blue” is the same as the prior “probability of you having red” and they are both ½. Assuming that, I could work out everything else, feed in the fact that we know I have a blue stone, and make an estimate of the chances that you do in fact have a red one.

We can definitely learn something from this exercise, and from knowing that I have in fact got a blue stone. For example, in the range of possible prior probabilities, we can rule out the one where all three stones are red – because I couldn’t have a blue one in that case! We could use this to update our prior probabilities, tilting them away from ½ each. We might also know something about the global population of red and blue stones, perhaps that there are more blue stones than red stones in the world, and feed that in. Or that the favourite colour of the person preparing the bag was red, so they might be more likely to put red stones in there than blue ones. Or alternatively they might save red ones for themselves. We might assume all bags contain the same mix, and update our prior belief based on a bunch of previous stone-picking experiments. It all gets a bit messy, judgement enters, and … oh no! What happened to scientific objectivity?

Setting aside the rigour of the evidence (for instance, how sure am I my stone is really blue and came from that bag?), Bayes’ theorem acknowledges that even if you have good evidence, the impact it has on your belief – perhaps better expressed as the credibility you place on any given theory – will depend on your prior assessment of the theories involved.

In practice this true in science whether scientists like admitting it or not - hence the “subconscious Bayes factor” in 5 sigma. We have well-established theories which have passed many tests, so a result claiming new physics amounts to an extraordinary claim, demanding extraordinary evidence.



It is also what humans do in general, and the mathematical tools of probability simply allow us to acknowledge that, and to make the exercise explicitly reproducible, which is surely a good scientific attribute.

For example, as a writer and head of a physics department, I get quite a few unsolicited communications about new theories of physics, often involving Einstein having been wrong, or the Higgs boson actually being a macaroon or something. I have a prior bias here, based on the enormous amount of existing evidence. Einstein might have been confused about the cosmological constant on occasion, but given prior evidence it is highly unlikely that the whole thrust of relativistic mechanics is up the spout. Likewise, I personally have quite a lot of evidence that the Higgs boson is consistent so far with being the fundamental Higgs of the Standard Model, and inconsistent with the macaroon theory. And the others. Jim Al-Khalili was being Bayesian when he promised to eat his boxer shorts if neutrinos travelled faster-than-light. He was correct.

Climate change is another good example. If you have a prior assumption that modern life is rubbish and technology is intrinsically evil, then you will place a high prior probability on carbon dioxide emissions dooming us all. On the other hand, if your prior bias is toward the idea that there is a massive plot by huge multinational environmental corporations, academics and hippies to deprive you of the right to drive the kids to school in a humvee, you will place a much lower weight on mounting evidence of anthropogenic climate change. If your prior was roughly neutral, you will by now be pretty convinced that we have a problem with global warming. In any case, anyone paying attention as evidence mounts would eventually converge on the right answer, whatever their prior – though it may come too late to affect the outcome, of course.

There is an exception to this, worth noting.

A prior assumption of zero probability can never be changed. Thus, for example, if you absolutely believe that the Earth is 5,000 years old or so, no amount of evidence can change your mind. If your unshakeable faith tells you there are only red stones, then the fact that I appear to have a blue one is simply god, or possibly satan, making a red stone look blue to test your faith. Just like he did with the fake fossils and the cosmic microwave background. No evidence will modify your prior belief. Your faith makes you impervious.



I guess Bayesian statistics provides a mathematical definition of a closed mind. Anyone with a prior of zero about something can never learn from any amount of evidence, because anything multiplied by zero is still zero.



On the more positive side, Bayes (who was an 18th century priest, by the way) allows us to acknowledge, and therefore somehow accommodate, our prejudice and bias, as well as the weight of prior evidence, and therefore, in my opinion, provides as close to a mathematical description of “the scientific method” as we’re likely to see.

But of course, I’m biased.

¹No, really, it does

Jon Butterworth has written a book about being involved in the discovery of the Higgs boson, Smashing Physics, available here . Some interesting events where you might be able to hear him talk about it etc are listed here. Also, Twitter.









