An American scientist visiting the home of Niels Bohr, the Nobel Prize-winning Danish physicist and refugee from Nazism who was a leading figure in the Manhattan Project, which produced the atomic bomb, was surprised to discover a horseshoe hanging over Bohr’s desk: “Surely you don’t believe the horseshoe will bring you good luck, Professor Bohr?” he asked. “After all, as a scientist . . .”

Bohr laughed. “I believe no such thing, my good friend. Not at all. I am scarcely likely to believe such foolish nonsense. However, I am told that a horseshoe will bring one good luck whether you believe it or not.”

Dominic Johnson, who tells this story, acknowledges that Bohr might have been joking. But the physicist’s response captured an important truth. Human beings never cease looking for a pattern in events that transcends the workings of cause and effect. No matter how much they may think their view of the world has been shaped by science, they cannot avoid thinking and acting as if their lives are subject to some kind of non-human oversight. As Johnson puts it, “Humans the world over find themselves, consciously or subconsciously, believing that we live in a just world or a moral universe, where people are supposed to get what they deserve. Our brains are wired such that we cannot help but search for meaning in the randomness of life.”

An evolutionary biologist trained at Oxford who also holds a doctorate in political science, Johnson believes that the need to find a more-than-natural meaning in natural events is universal – “a ubiquitous phenomenon of human nature” – and performs a vital role in maintaining order in society. Extending far beyond cultures shaped by monotheism, it “spans cultures across the globe and every historical period, from indigenous tribal societies . . . to modern world religions – and includes atheists, too”.

Reward and punishment may not emanate from a single omnipotent deity, as imagined in Western societies. Justice may be dispensed by a vast unseen army of gods, angels, demons and ghosts, or else by an impersonal cosmic process that rewards good deeds and punishes wrongdoing, as in the Hindu and Buddhist conception of karma. But some kind of moral order beyond any human agency seems to be demanded by the human mind, and this sense that our actions are overseen and judged from beyond the natural world serves a definite evolutionary role. Belief in supernatural reward and punishment promotes social co-operation in a way nothing else can match. The belief that we live under some kind of supernatural guidance is not a relic of superstition that might some day be left behind but an evolutionary adaptation that goes with being human.

It’s a conclusion that is anathema to the current generation of atheists – Richard Dawkins, Daniel Dennett, Sam Harris and others – for whom religion is a poisonous concoction of lies and delusion. These “new atheists” are simple souls. In their view, which derives from rationalist philosophy and not from evolutionary theory, the human mind is a faculty that seeks an accurate representation of the world. This leaves them with something of a problem. Why are most human beings, everywhere and at all times, so wedded to some version of religion? It can only be that their minds have been deformed by malignant priests and devilish power elites. Atheists have always been drawn to demonology of this kind; otherwise, they cannot account for the ­persistence of the beliefs they denounce as poisonously irrational. The inveterate human inclination to religion is, in effect, the atheist problem of evil.

But what if belief in the supernatural is natural for human beings? For anyone who takes the idea of evolution seriously, religions are not intellectual errors, but ­adaptations to the experience of living in an uncertain and hazardous world. What is needed – and still largely lacking – is a perspective in which religion is understood as an inexhaustibly complex variety of beliefs and practices that have evolved to meet enduring human needs.

God Is Watching You is a wide-ranging and consistently interesting attempt to redress this default. Vividly written and packed with arresting examples, the book shows how belief in supernatural sanctions can temper short-term self-interest and foster social solidarity. An important piece of evidence is a groundbreaking study by two psychologists, Azim Shariff and Ara Norenzayan, which describes the “dictator game”, an experiment in game theory in which subjects are given a sum of money and the freedom to split it as they please with an anonymous person. As the choice is secret and there are no negative consequences, the response of Homo economicus would be to keep all the money for himself. In fact, few people respond in this way. Numerous studies have shown people giving away roughly half of the money, and those who have been exposed to religious concepts and beliefs usually hand out more.

Interestingly, a follow-up experiment found that fear of supernatural punishment was more effective as an inhibitor of selfish behaviour than hope of supernatural reward. A deity watching over our wrongdoing makes for a claustrophobic picture of the world, and the idea that people are best controlled by fear suggests an unattractive view of the human animal. Yet it does seem that belief in a punitive God can act as a remarkably powerful curb on individual behaviour for the sake of social order. Many will object that the morality that supernatural beliefs enforce is often highly repressive. No doubt this is true, yet it is hard to see what grounds new atheists can have for denying that illiberal moralities can have evolutionary value. After all, very few societies have been liberal for long. Liberal values may turn out to be no more than a blip in the evolutionary process. Though the current generation of unbelievers chooses to forget the fact, this was the conclusion of many earlier atheist thinkers – communists, positivists and assorted social engineers – who flirted with evolutionary ethics.

Citing other experimental studies showing similar results, Johnson develops a forceful case for the evolutionary role of religion in cementing social co-operation. In doing so, he has added a chapter to the long-running debate about how science relates to religion. Inevitably, his argument is less than watertight. For one thing, by no means all religions focus on a supernatural being whose principal concern is with the punishment of wrongdoing. In the ancient Greek pantheon the gods may be as fickle and unpredictable as human beings, if not more so: Hermes, the patron of thieves, merchants and orators, was celebrated for using trickery to outwit people and other gods. The Roman and Babylonian civilisations promoted many practices involving the supernatural but their gods were not especially moral or concerned to penalise those who violated canons of good behaviour. Johnson is aware of this problem:

If supernatural punishment is about reducing selfishness and promoting good behaviour and co-operation, then a remaining puzzle is why there are some supernatural agents who do not merely fail to punish, but punish the wrong people. Why, for example, were some of the Greek gods so jealous, vengeful and vindictive? Why in the Book of Job did a perfectly good God exact apparently unjustified and arbitrary punishment on an innocent victim? Why are there even supernatural agents that work against each other? God and Satan is one obvious example, but the phenomenon is found elsewhere. The Greeks could appeal to one god for protection against another.

Though he concedes that these examples “seem to cut against the grain” of his theoretical framework, Johnson believes they are outliers. “What matters is the overall trend . . . Capricious gods are no more of a problem to a theory of supernatural punishment than the presence of corrupt politicians to a theory of democratic government. With enough selection, or enough elections, the things that matter shine through.” In other words, evolution will ensure that the religions that prevail will be those that enable social co-operation by promoting a belief in supernatural punishment. The trouble is that this is a blank cheque, rather than a falsifiable hypothesis. Understanding religion as an evolutionary adaptation is unavoidable if you think of the human animal in Darwinian terms. Asserting that evolution favours religions to which divine punishment is central is another matter. No one has ever identified a mechanism of selection amongst religions, and it is unclear whether any such mechanism would operate on individuals, social groups or some combination of the two. These are problems that beset all theories of cultural evolution. In the end, such theories may be not much more than misplaced analogies and misguided metaphors.

Johnson is on stronger ground in suggesting that a need to find meaningful patterns in random events may be hard-wired in human beings. Here the historical record of atheism is instructive. Johnson devotes a longish chapter to what he calls “the problem of atheists”, arguing that, like the rest of humankind, they are “inclined to supernatural thinking”, which lives on in them in the form of “superstitious beliefs and behaviours”. This may be so, but it is not the most important way in which atheists have continued to be governed by the needs that religions have evolved to satisfy. Practically without exception, the atheist movements of the past few centuries testify to a demand for meaning that has led them to replicate many of the patterns of thinking distinctive of monotheism, and more particularly of Christianity.

For Christians, human history isn’t an endless succession of cycles – as it was for the ancient Greek and Romans, for example – but a story, and one of a distinctive kind. Unlike practitioners of polytheism, who seek and find meaning in other ways, Christians have found sense in life through a mythical narrative in which humankind is struggling towards redemption. It is a myth that infuses the imagination of countless people who imagine they have left religion behind. The secular style of modern thinking is deceptive. Marxist and liberal ideas of “alienation” and “revolution”, “the march of humanity” and “the progress of civilisation” are redemptive myths in disguise.

For some, atheism may be no more than a fundamental lack of interest in the concepts and practices of religion. But as an organised movement, atheism has always been a surrogate faith. Evangelical atheism is the faith that mass conversion to godlessness can transform the world. This is a fantasy. If the history of the past few centuries is any guide,

a godless world would be as prone to savage conflicts as the world has always been. Still, the belief that without religion human life would be vastly improved sustains and consoles many a needy unbeliever – which confirms the essentially religious character of atheism as a movement.

Atheism need not be an evangelical cult. Here and there one finds thinkers who have truly left redemptive myths behind. The American journalist and iconoclast H L Mencken was a rambunctious atheist who delighted in lambasting religious believers; but he did so in a spirit of mockery, not out of any interest in converting them into unbelievers. Wisely, he did not care what others believed. Rather than lamenting the fact of incurable human irrationality, he preferred to laugh at the spectacle it presents. If monotheism was, for Mencken, an amusing exhibition of human folly, one suspects he would have found the new atheism just as entertaining.

Certainly there is an element of comedy in the new atheist mix of proselytising Darwinism and ardent rationalism. There is no way in which a model of the mind inherited from Descartes and other rationalist philosophers can be squared with the findings of evolutionary biology. If you follow Darwin in thinking of human beings as animals that have evolved under the pressures of natural selection, you cannot think that our minds are primed to seek out truth. Rather, our ruling imperative will be survival, and any belief that promotes this will have a powerful attraction. This may be why we are so anxious to discern a pattern in the drift of events. If there is none, our future will depend largely on chance – a dispiriting prospect. The belief that our lives unfold under some kind of supernatural direction offers a way out, and if this faith enables us to live through disaster, that it may be groundless is irrelevant. From an evolutionary perspective, irrational belief isn’t an incidental flaw in human beings. It has made us what we are. In that case, why demonise religion?

Johnson concludes that dispensing with religion may be an unwise move. “To assume that this complicated old machine we have found in our evolutionary garage is of no use, and to fling it on to the scrapheap of history seems rather hasty,” he writes. “We might need it later.” In fact, the logic of his argument points in a very different direction. If religion is an evolutionary adaptation, converting humankind from it isn’t so much unwise as simply impossible.

The irony of the new atheism is that it is pre-Darwinian. Making sense from the chaos of human events, religions provide something that science cannot offer, but which most people still desperately want. Accordingly, the new atheists have turned science into a religion: a gospel of enlightenment that can deliver the world from darkness. Possessed by this ersatz creed, which has all of the flaws of conventional religion and none of the saving graces, our crusading unbelievers are comically oblivious of their own need for faith. It takes a genuine scientist like Bohr to see and state the obvious.

God Is Watching You: How the Fear of God Makes Us Human by Dominic Johnson is published by Oxford University Press (304pp, £18.99)