In the 1920s, one of Carl Jung’s female patients proved particularly frustrating to him – notwithstanding her ‘excellent education’ and ‘highly polished Cartesian rationalism’. She was ‘psychologically inaccessible’, the Swiss psychiatrist later wrote in his Synchronicity: An Acausal Connecting Principle (1960), by which he meant that she wasn’t accepting his pseudo-scientific methods.

To better understand her subconscious mind, Jung had her recount her recent dreams. She told him that, the night before, she had dreamed that she’d been given a golden scarab as a piece of jewellery. As she was describing the dream, there was a tapping on the window and Jung turned around. ‘I opened the window immediately and caught the insect in the air as it flew in,’ he wrote. ‘It was a scarabaeid beetle, or common rose-chafer (cetonia aurata), whose gold-green colour most nearly resembles that of a golden scarab.’ Jung knew this was just what his skeptical patient needed to see. ‘I handed the beetle to my patient with the words, “Here is your scarab.” This experience punctured the desired hole in her rationalism and broke the ice of her intellectual resistance. The treatment could now be continued with satisfactory results.’

Jung called this an instance of ‘synchronicity’, a concept whose application to psychology he developed with the Austrian-born theoretical physicist Wolfgang Pauli to describe the way that seemingly unrelated events can in fact be significantly related and held together by an unseen force – in this case, his patient’s dream and shared reality coalescing with the appearance of the scarab beetle. Jung believed in an unus mundus, or a unitary world, in which there is no separation between mind and matter. Everything is connected; every event has a reason behind it. It spurred his belief in even wilder ideas such as telepathy, and fed his concept of the ‘collective unconscious’, for which he claimed there were certain universal ideas, beliefs and archetypes implicitly understood by everyone from birth.

The Austrian biologist Paul Kammerer, a contemporary of Jung’s, built on ‘synchronicity’ with his theory of ‘seriality’, which says that coincidences are a basic force of the Universe, like gravity. Albert Einstein, always pushing the boundaries between faith and reason, found the quasi-spiritual idea of seriality intriguing, and is rumoured to have called it ‘original and by no means absurd’.

Today, nearly all scientists say that coincidences are just that: coincidences – void of greater meaning. Yet, they’re something we all experience, and with a frequency that is uniform across age, sex, country, job, even education level. Those who believe that they’ve had a ‘meaningful coincidence’ in their lives experience a collision of events so remarkable and unlikely that they chose to ascribe a form of grander meaning to the occurrence, via fate or divinity or existential importance. One of the most commonly experienced ‘meaningful coincidences’ is to think of your friend for the first time in a long while only to have her telephone you that instant. Any self-respecting statistician would say that if you tracked the number of times you thought of any friend, and the number of times you had that friend immediately ring you, you’d find the link to be statistically insignificant. But it is not necessarily irrational to attribute grander significance to this occurrence. To those who believe in meaningful coincidences, statistical insignificance does not undermine an event’s causality or importance. To them, just because something could happen doesn’t mean it wasn’t also fated to happen.

It’s a mindset that applies equally to our habit of weaving relationships between coincidental events into epic myths, religious stories, even conspiracy theories. Longwinding, Dickensian stories of interconnected coincidences leading to a cathartic conclusion can provide us with a sense of meaning, of life holding subtler, unseen mysteries that make even our suffering worthwhile – as if our lives were really a series of sophisticated, interconnecting puzzle pieces. This largely explains the seductiveness of most mainstream religions as well: a divine hand orchestrating our lives is a particularly comforting notion, even if, scientifically, there’s little to lend credence to such beliefs.

But just because we might ‘know’ that meaningful coincidences don’t really exist doesn’t mean that they don’t still move us. The poet John Keats in 1817 accused Isaac Newton of trying to ‘unweave the rainbow’, by which he meant that Newton was attempting to take the magic out of life by paring it down to its scientific basis. The young poet might have been wishful in his thinking, but such a statement also raises the question of how we should grapple with mysteries – with or without a belief in a greater meaning to life? Even if every possible coincidence could be scientifically explained, we shouldn’t necessarily discount its importance. You can watch a movie or read a novel, and be at once aware of its nonreality while also being moved by it. Must these ideas therefore be incompatible? Indeed, might the continued belief in meaningful coincidences even be rational and necessary to our experience of existing in the world? And, is a belief in meaningful coincidences something vital to our survival as humans?

After the so-called Freud Wars starting in the 1970s, led by the American essayist Frederick Crews, any orthodox adherence to Freudian or Jungian ideas has since been frowned upon in the mainstream scientific community. Statistical and evolutionary arguments against notions of synchronicity, seriality and meaningful coincidences at large have come to seem ironclad, and the existential aspects of coincidence have been wholly discounted. Those who do believe in meaningful coincidences also haven’t been doing many favours for themselves. People who strongly believe in the paranormal and in conspiracy theories, for instance, tend to be significantly worse at probabilistic and statistical reasoning than those who don’t believe in them, according to studies from the University of Bristol and Goldsmiths, University of London, respectively. In truth, most of us are surprisingly poor at gauging the probabilities of events, so when we receive that phone call from the friend we’re thinking of, we’re prone to ascribe to it a significance disproportionate to its relative commonness.

A good example of our lack of statistical logic is when gamblers interpret a run of black or red numbers in roulette as meaningful, in spite of the fact that each time one colour comes up, the next spin has exactly the same 50 per cent probability of landing on black or red. Or, take ‘the birthday problem’, which simply asks: how many people would you need to get into the same room in order to statistically assure that at least two share the exact birth month and day? Given that there are 365 days in a non-leap year, and that most people you know probably don’t have the same birthday, you might reasonably suppose that you’d need quite a high number to find an exact match. Hundreds, perhaps, and even then you’d be lucky to find two people with the same birth month and day. Statistically, however, you need only 23 people in the room for a greater than 50 per cent (hence ‘statistically probable’) chance of finding two people with the exact same birth month and day. For a 99.9 per cent chance, you need only 70 people.

We ascribe exceptional meaning to what we perceive as exceptionally low-probability events, but they’re often not as low-probability as we think. And, even if they are unlikely, the most unlikely events are – with 7 billion people on Earth – actually relatively common, thanks to the so-called law of truly large numbers, the statistical adage of Frederick Mosteller and Persi Diaconis, in which a big sample size will eventually lead to essentially any result. Many people have survived being struck by lightning (even multiple times). Many have won the lottery (even multiple times). Plus, we’re culturally trained to see meaning in intrinsically meaningless events: Thomas Jefferson and John Adams died hours apart on the 50th anniversary of the Declaration of Independence; Mark Twain was born and died on days in which Halley’s Comet could be viewed from Earth. There’s statistically nothing further to be derived from these events, but we talk about them regardless as evidence of a greater, often spiritual, meaning. Every coincidence can be statistically explained. Even Plutarch understood this. Writing in ‘The Life of Sertorius’, a volume in his Parallel Lives series (1st century CE), he noted: ‘It is no great wonder if in the long process of time, while fortune takes her course hither and thither, numerous coincidences should spontaneously occur.’

Concepts such as synchronicity are ‘very seductive, because mysteries always are’

Irrespective of the mathematics of coincidence, there are still psychology specialists unwilling to give up on adapting theories of synchronicity and seriality. Bernard Beitman, a psychiatrist in Virginia and the author of the bestselling Connecting with Coincidence: The New Science for Using Synchronicity and Serendipity in Your Life (2016), believes that meaningful coincidences both exist and can be proven. One story he likes to tell about coincidence is extremely personal. He was 31 years old and living in San Francisco. One night, at about 11pm, he began violently choking over the sink. He hadn’t been eating or drinking, and the attack seemed to come out of nowhere. He drank some water, recovered, and then went to bed wondering what had spurred his choking fit. The next morning, he received a call from his brother, who told him that their father had died at two in the morning in Connecticut – which, because of the three-hour time difference, made it the same time as when Beitman had begun choking. His father had died choking to death on his own blood. Beitman found this pair of events incredibly significant, but he couldn’t find a sufficiently scientific explanation for them other than random chance. So he developed his own explanation, taking, in part, from Jung’s idea of synchronicity.

Beitman called this experience ‘simulpathity’ – or the experience of another person’s suffering, even from afar. He has since tried to provide a scientific basis for other Jungian ideas such as serendipity and seriality, and his overarching view is inherently Jungian, invoking the unus mundus while adding a distinctly New Age twist. ‘I use a term called “the psycho-sphere”,’ Beitman told me, ‘by which I mean only that mental atmosphere that surrounds Earth, in which we are immersed right here.’ Admittedly, trying to find a scientific basis for believing in meaningful coincidence is not a widespread pursuit (Iris Bell, a psychiatrist at the University of Arizona, with whom Beitman has co-written a book, is another rare exception). Statisticians still push back.

‘I find it very surprising, given the incontrovertible mathematical arguments,’ says David Hand, a prominent statistician and emeritus professor of mathematics at Imperial College London. ‘If you look at the relative numbers – especially of statisticians and probabilists, who are expert in such matters – you will find a vanishingly small percentage who would agree with concepts like synchronicity.’ But, perhaps, Hand reasons, the popularity of Beitman’s ideas isn’t all that surprising given human nature. ‘It’s very seductive, because mysteries always are.’

In fact, Beitman has taken a clever tack in recognising that just because something is random doesn’t also mean it’s not significant. It’s a similar argument that an atheist will eventually face when debating with a religious person who believes that God pulls the strings of life. Just because the atheist can use the law of truly large numbers to statistically prove that every event that happens within the boundaries of the physically conceivable world is possible without a god, this proof doesn’t also mean that those chance events are not somehow hiding meaning. (Or indeed, that some magical string-puller might not be behind at least some of them)

Meaning cannot be quantified or even clearly and routinely identified. The difficulty that Beitman faces is in trying to make meaningful coincidence into a scientific concept. Like a religious person, the greatest asset to believing in meaningful coincidence is that you cannot prove that something is devoid of meaning since ‘meaning’ is not scientifically testable. Where Beitman is most successful – even rational – is when he shows how experiencing a coincidence can encourage psychological shifts. He tells the story of a patient who told him that she was letting her abusive husband return to living with her. But before she went to get him from the airport, she received a phone call. The woman on the other end had the wrong number; yet, for reasons unexplained, they continued to talk, and it came out that the other woman had an abusive boyfriend. The other woman sounded fearful and unstable, and after hanging up the married woman decided to separate from her husband after all. This, Beitman says, is a meaningful coincidence that fundamentally altered her psychology and outlook on life. It is also why meaningful coincidences are so important. With the exception perhaps of the chilliest of rationalists, these types of events tend to have deeply visceral, sometimes life-changing effects.

Cynically, one could reason that trying to add a scientific patina to the belief in meaningful coincidence is driven by greed: there will always be money to be made in writing books about or giving talks that exploit people’s desire to see chance coincidences as significant. We want evidence of a hidden meaning in life – and we’re willing to pay for it. Beitman might be one of the few trying to provide an academic spin to meaningful coincidences, but there are literally thousands of books and movies about the beauty, significance and importance of coincidences (not to mention how they’re used to win religious adherents). Coincidences, write Yitta Halberstam and Judith Leventhal in their bestselling book Small Miracles: Extraordinary Coincidences From Everyday Life (1997), show ‘the rich promise of a bounteous universe and the splendour lying dormant within your soul. Coincidences are everywhere and can happen any time. When your soul is ready, they will come. All that is required is that you open your heart.’ This is obviously just a bit of feelgood hocus pocus. But it’s not actively harmful either, other than in taking advantage of people’s willingness to pay to hear what they want to hear.

Where a belief in meaningful coincidences can become dangerous, however, is when it begins to impair your judgment. For one, it can make you think illogically. For instance, ‘overfitting’ occurs when you fit your belief model to the noise rather than the signal, like suddenly ‘seeing’ UFOs after your friend has been talking about them. ‘Hidden meanings’ come out of the Jungian and Freudian ideas of the collective and individual unconscious, in which ‘slips’ of words or phrases are viewed not merely as linguistic errors but as pointing towards a greater subconscious, psychological meaning that’s not really there. Or the aforementioned ‘gambling fallacy’, which occurs when you begin to see seemingly hidden patterns in outcomes that result only by chance.

But while these might seem like small-time issues – UFOs, Freudian slips – there are sometimes larger questions at play. For instance, when the anti-vaccination movement was embraced in the United States, children died or had their health imperilled by parents who aligned rising rates of autism in children to the rising number of vaccines being given. It’s true that autism tends to be diagnosed around the same time that vaccinations are administered, but anti-vaxxers confused correlation with causality, relying, in part, on a belief in meaningful coincidence. Something has to be making certain children around the age of vaccinations get sick, they thought. But as study after study has shown, there’s no stock in the claim that vaccines make children autistic or even sick. Scientists, therefore, need to help ‘people to make decisions on the basis of concrete evidence, not half-baked pseudoscience,’ says Hand, the statistician. ‘Scientists have a public duty to help to disperse the mists of confusion.’

A belief in meaningful coincidence is, from an existential perspective, surprisingly rational

‘When we experience coincidences, we experience a pattern of events, sometimes perceived to be very rare, and that are surprising to us, that feel like they are caused by something,’ adds Magda Osman, who teaches experimental psychology at Queen Mary University of London. But just because we can’t say why, exactly, certain events happen at certain times doesn’t give them meaning, she says. ‘Coincidences are just an inevitable part of our cognitive system. That is it.’

What so much of the question around meaningful coincidence comes down to is how you choose to fill the vacuum of life’s mysteries. Is the realm of the unknown a place of spirituality and existential significance for you, or does the world remain entirely material?

Beneath the statistical incorrectness, beneath any economic ploys, beneath even the potentially grave errors that can result, a belief in meaningful coincidence is, from an existential perspective, surprisingly rational. If your father were to choke to death across the country at the same time that you felt a phantom choking, you might know, intellectually, that there was no mysterious, invisible connection at play. But, if you did let your mind wander to that possibility, it would allow you a new way to grieve your father’s death, giving you a sense of intimacy or a fatalist understanding of events. Beitman claims that science is ‘fairly flexible’, which seems like a red flag to serious researchers. But behind such a statement – and motivating the millions of people who buy books on, watch movies about, or have ever thought about what the many links between the events in one’s life mean – is the sometimes-necessary need to fabricate meaning.

We do this in infinite ways, not least via the apparatus of religion, but one way of finding meaning is to marry the mental and material worlds, signing up to what Jung called the ‘collective unconscious’ – an invisible realm that binds together our lives. It’s a belief that’s scientifically disproven, of course, and it’s flawed from its start; but, like so many other non-scientific beliefs, from a psychological angle, it makes some sense. There is, after all, a difference between statistical significance and human significance – one does not always dictate the other. Wrong and right all at once. A beetle is just a beetle until you decide that it’s not.