We all have lucky rituals or charms but why do we see meaningful patterns or connections in random or meaningless data, asks Michael Blastland.

So the football highlights are on television and you're sitting down to watch them in your lucky socks.

Yes, that's after the whistle for the end of the game has actually blown, as if your socks can change the direction of causality (among multiple other violations of the known laws of physics).

The only numbers you care about are the score and the match statistics that show you won. I know you don't care much for grand statistical principles right now. But I must warn you - for I have your welfare at heart - that aside from any offence against hygiene and basic rationality, you and your lucky socks are also committing a grave statistical error.

This error is so fundamental that it's known as the Type I error. Though the Type II error is pretty fundamental too. You might even be suffering from apophenia, of which more later.

And a couple more sock shocks while you're still paying attention. Firstly, your behaviour is allegedly shared with pigeons. A discovery made in 1947, though only in principle as pigeons don't wear socks. Secondly, it's the kind of mistake that might even be good for you.

Accidental association

So, socks. People have lucky socks, among other things, for sporting and other occasions. What's statistical about that? Who cares? Tell me about the pigeons first, you say, then I'll read about the stats.

Fair enough. In 1947, the famous behavioural psychologist BF Skinner stuck a few half-starved pigeons in a box and put food pellets down a chute at random intervals. The pigeons, said Skinner, began to think that whatever they did when the food arrived somehow caused the food to arrive, so they did it often.

Image caption In tests pigeons thought they caused food to arrive

"One bird was conditioned to turn counter-clockwise about the cage, making two or three turns between reinforcements," he wrote. "Another repeatedly thrust its head into one of the upper corners of the cage. A third developed a 'tossing' response.

"The experiment might be said to demonstrate a sort of superstition. The bird behaves as if there were a causal relation between its behaviour and the presentation of food, although such a relation is lacking."

He called this "adventitious reinforcement". Some accidental association between an outcome and whatever you are doing at the time - wearing those socks on the day of a huge win - becomes established in your mind as a real relationship.

It has to be said that some dispute Skinner's analysis, and since he also developed a pigeon-guided missile during World War II, you can tell he liked provocative ideas - and maybe not pigeons.

This tendency to make connections where none exist - sorry to break it to you like that - has also been called apophenia, which Wikipedia defines as: "The experience of seeing meaningful patterns or connections in random or meaningless data."

Connection illusion

Which brings us to the link with stats. In statistical terms, it's all about over-interpreting randomness. The food can come at any time, but pigeons don't deal well with randomness, so they invent a cause for random events. Same for people: we don't like to feel that the relationship between us and our team's results is random, so we might seek a near-to-hand, or -foot, association: "Ah! That must have happened because..."

We did not evolve a Baloney Detection Network in the brain to distinguish between true and false patterns Michael Shermer, Skeptic magazin

One of the biggest problems in stats is trying to work out whether A really causes B. Or whether B just happened by random chance or for some other reason. The starting point for investigating these problems is to assume there's no link between A and B, what's known as the null hypothesis. Think of it in our case as the nil hypothesis - the hypothesis is that your socks won't make England score. We then reject the nil/null hypothesis if we find evidence that your lucky socks really do make England score.

If we reject the nil/null hypothesis falsely - still with me - and decide the socks make a difference when really they don't, we have committed a Type I error. We think something's afoot when it isn't.

The other type, the Type II error, is to think there's nothing going on when there is. Unbeknownst to you, your socks have been the determinant of Estonian volleyball for the last decade but you stupidly dismissed the possibility. So the truth about your socks is the same as the most fundamental problem in stats.

For a final twist, and another reckless leap of discipline, all this might even have been beneficial to your survival. In an article a few years ago in Scientific American called Patternicity: Finding Meaningful Patterns in Meaningless Noise, Michael Shermer wrote:

"Our brains are… evolved pattern-recognition machines that connect the dots and create meaning out of the patterns that we think we see in nature. Sometimes A really is connected to B; sometimes it is not. When it is, we have learned something valuable about the environment from which we can make predictions that aid in survival and reproduction… Unfortunately, we did not evolve a Baloney Detection Network in the brain to distinguish between true and false patterns."

Which is how we survived and evolved well enough to reach that advanced point of civilisation known as the lucky socks period.

In the end, Brian, when all's said and done, even if our hyperactive pattern recognition systems create only the illusion of connection and control over a sporting event, well, so what? It's a hopeful illusion that might even be the pinnacle of evolution. Scores nil for stats, though.

You can try your instincts for pattern recognition, Type I error and apophenia here: