Ben Goldacre

Saturday April 21, 2007

The Guardian

So this week in the papers a man was allergic to his own hair, bee colonies were collapsing because of mobile phones, and more. Speaking as the veteran of a great many squabbles, on MMR, phone masts, drugs, and more, I can tell you: facts are not entirely welcome. When all the evidence goes against someone’s beliefs, they will tell you, quite plainly, that they just know it to be true. They sense it. They intuit it. Nobody will ever listen to an explanation of why intuitions can be flawed – presumably because their intuitions have told them not to.

But we have an innate human ability to make something out of nothing. We see shapes in the clouds and a man in the moon; gamblers are convinced that they have “runs of luck”; we can take a perfectly cheerful heavy metal record, play it backwards, and hear hidden messages about satan. Our ability to spot patterns is what allows us to make sense of the world but sometimes, in our eagerness, we can mistakenly spot patterns where none exist.

In science, if you want to appreciate a phenomenon, it is often best to reduce it to its simplest and most controlled form. There is a prevalent belief among sporting types that sportsmen, like gamblers (except even more plausibly) have “runs of luck”. People ascribe this to confidence, “getting your eye in”, “warming up”, or more, and while it might exist somewhere, statisticians have looked in various places and found no relationship between, say, hitting a home run in one shot, and then hitting a home run in the next.

Because the “winning streak” is such a prevalent belief, it is an excellent model for looking at how we perceive random sequences of events, and this was used by a social psychologist called Thomas Gilovich in a classic experiment. He took basketball fans and showed them a random sequence of X’s and O’s â€“ telling them that they represented hits and misses in a basketball game â€“ and then asked them if they thought the sequences demonstrated streak shooting.

Here is a perfectly random sequence of figures from that experiment. You could think of it as being generated by a coin being flipped (I can explain why it’s random if you want but it’s a bit boring, essentially there is no correlation between one outcome and the next, and the number of adjacent figures with the same outcome â€“ xx or oo â€“ is the same as the number of adjacent figures with different outcomes â€“ xo or ox). Here’s that random sequence:

OXXXOXXXOXXOOOXOOXXOO

The subjects in the experiment, when shown this entirely random sequence, were convinced that it exemplified “streak shooting”, or “runs of luck”. It’s easy to see why, if you look again: 6 of the first 8 shots were hits. No, wait: 8 of the first 11 shots were hits. I agree: no way does that look random. But it is.

What this ingenious experiment shows is just how bad we are at correctly identifying random sequences. We are wrong about what they should look like: we expect too much alternation, and to us, even truly random sequences seem somehow too lumpy and ordered.

Why is this important? Because it shows that our intuitions about the most basic observation of all, from which all others follow â€“ our abilities to distinguish an actual pattern, from mere random background noise â€“ are deeply, deeply flawed.

You cannot sense whether a pill improves intelligence, or cures the common cold, or whether MMR causes autism. Your tiny, beautiful ingot of human experience in the world does not present you with sufficient information to spot patterns on that scale: it’s like looking at the ceiling of the Sistine chapel with one eye through a very long cardboard tube.

Intuitions are great shortcuts. They’re very valuable for lots of things in the social domain: deciding if your girlfriend is cheating on you, perhaps, or whether a business partner is trustworthy. But for mathematical issues, or assessing causal relationships, intuitions suffer from inaccuracies, misfires, and oversensitivity. The challenge, perhaps, is to work out which tools to use where: because trying to be “scientific” about your relationship is as stupid as following an intuition about the risks and benefits of a treatment.