The idea of atheism has never been as clear as you might expect. Etymologically, it ought to refer to the idea that there is no such thing as God, or an attitude of indifference or defiance even if there is. In practice, however, it has usually been used by religious sectarians to hit out at anyone suspected of doctrinal deviancy, or – in one version of a message received by Moses – those who “go a-whoring after strange gods”. Socrates, for example, was denounced as atheos by his fellow Athenians, though they knew he was a believer in his way, and when he tried to defend himself he felt, according to Plato, as if he was “fighting with shadows.” When St Paul talked about “atheists” (“strangers … without God in the world”) he did not mean unbelievers, but traditionalists who had not heeded the gospel of Christ; and Christians got a dose of their own semantic medicine when they found themselves arraigned as “atheists” under the provisions of Roman law. Socrates

There is a parallel with anarchism – a term which, until its adoption by Pierre Proudhon in the 1840s, was always used to disparage rather than describe. Take the contrarian poet Percy Shelley: today he might well be classified as an anarchist, but he himself would have repudiated the description. Following the Peterloo massacre in 1819, he turned on Castlereagh and the British government – “I met Murder on the way, he had a mask like Castlereagh” – but as far as he was concerned the root of political evil was anarchy: “anarchy … on a white horse … trampling to a mire of blood the adoring multitude.”

But if anarchy was beyond the pale for Shelley, atheism was not. His pamphlet on “The Necessity of Atheism,” published anonymously in 1811 when he was 18, got him expelled from Oxford and disowned by his family, but he stood by it all the same. He may not have been the first atheist to come out of the closet, but he was the first to flourish the title with bravado and panache. On the other hand there was less to his atheism than meets the eye. “It is a good word of abuse,” he said, and he deployed it to advertise his revulsion from the Christian idea of a god who created the world and established the distinction between good and evil. But strictly speaking he was not so much an atheist as a pagan theist. His denial of God, he explained, “must be understood solely to affect a creative deity,” while the “hypothesis of a pervading spirit co-eternal with the universe remains unshaken.” In reflective moments he preferred to call himself a deist.

If the world’s first celebrity atheist was a deist then the word “atheism” seems to be in trouble. Hence the rise of the term “new atheism” to distinguish atheists who really mean business from those who prefer to hedge their bets. Like “atheism” itself, however, “new atheism” began life with negative connotations. It can be traced back to the 17th century, when it – or rather its French equivalent – was used to alert Christians to the threat of Spinozism. But nouvel athéisme was itself a dark phrase, since Spinoza believed passionately in something called God, though he shocked the orthodox by identifying it with nature as a whole rather than a transcendent supernatural agency. During the 19th century, as Spinoza came to be viewed as a pious mystic rather than a raucous infidel, the “new atheist” tag was transferred first to proponents of the mutability of species, then to Auguste Comte and the positivists, followed by the indomitable secularists Harriet Martineau and George Holyoake, Spencerian evolutionists and Darwinian natural-selectionists, and eventually Friedrich Nietzsche and his enigmatic hero Zarathustra. Spinoza

New atheism was born again at the beginning of the 21st century, and some people think it has dealt a final blow to religion in all its forms. The God hypothesis has been spelt out with perfect clarity, apparently, and anyone capable of following logical and scientific arguments can see that it has no merit at all. Religion must therefore be consigned – like the miasmic theory of disease or the phlogiston theory of combustion – to a museum of intellectual lost causes.

Some of us however – including many who regard ourselves as non-believers – suspect that the new new atheism forces the pace, distorts the issues, and underestimates the intelligence of its enemies. If the older versions of atheism – from Moses and Socrates to Shelley and Nietzsche – were less straightforward than they might have been, the reason may be the complexity of religious phenomena rather than the obtuseness of those who sought to describe them. The difficulty is that people may commit themselves to a religion without buying into any particular theory as to what does or does not exist: they are simply throwing in their lot with some historic community, identified not by doctrines but by rituals, stories and a shared sense of the sacred. Religion as it enters the lives of many believers will not be damaged by a demonstration that it is not much good as science, any more than poetry will be threatened by the collapse of literary theory, or capitalism by a refutation of neoclassical economics. We atheists should not assume that theory always gets the last laugh.

No one has understood the untheoretical aspects of religion better than the American philosopher William James. By background James was a Darwinian scientist, and after graduating in medicine in the 1860s he taught natural history at Harvard before switching to psychology and becoming the anti-metaphysical pragmatist he is remembered as today. In 1899 he agreed to give the Gifford lectures on religion in Edinburgh; but he soon began to have misgivings. He was not unsympathetic to religion, but he was so far removed from scriptural orthodoxy that he was afraid his approach would cause offence. He tried to cancel and then needed two postponements before venturing over to Scotland in 1901. Shelley

In the event the lectures were a triumph, and the book that grew out of them – Varieties of Religious Experience – met with extraordinary popular success. Working through piles of thank-you letters back on his New Hampshire farm, James noted that “God’s enemies and his friends both find fuel for their fires in my pages.” This was exactly the result he had wanted. He was not interested in provoking a showdown between self-appointed defenders of religion and self-appointed nemeses: it was not so easy to draw a clear line, he thought, and he preferred to provide enlightened comfort to each side or – what comes to much the same – to unsettle both of them equally.

Despite his Darwinism, James was impatient with the all-purpose “Darwinising”, as he called it, of scientific colleagues like TH Huxley or Ernst Haeckel. He hated the belligerent secularism that treats religion as a childish superstition which we will all put behind us once we reach the age of reason. For one thing, the idea of superstition is itself steeped in religiosity: like “heresy”, “idolatry”, “apostasy”, “blasphemy” or indeed “atheism”, it started life as a word for deviations from true faith, and the first self-declared enemies of superstition were not enlightened scientists but inquisitorial bigots. For another, not all believers are gullible fools, and intelligent religiosity might have more in common with intelligent infidelity than with ignorant faith. And in any case, religion for James was more a matter of subconscious experience than explicit doctrine. “Feeling is the deeper source of religion,” he wrote, and “philosophic and theological formulas are secondary products, like translations of a text into another tongue.” Philosophical theologians who tried to “construct religious objects out of the resources of logical reason” were missing the point, and chest-thumping atheists who tried to refute these intellectual constructions only compounded the error.

James liked to define religion by contrast: it was the opposite, he suggested, of the smug facetiousness and cackling je m’en fichisme cultivated by 18th-century philosophes like Voltaire, who treated any display of tenderness or solemnity as a sign of weakness or folly. But most of us have a capacity for respectful attentiveness, and we can, on occasion, “close our mouths and be as nothing.” Anyone with the courage to say “hush” to “vain chatter and smart wit” – anyone who could prefer “gravity” to “pertness” – was, James thought, ready for religious experience. Becoming religious was like falling in love, he said: not a process of intellectual persuasion, but not a delusion either, and it lent new aspects to the world, “an enchantment which is not logically deducible from anything else.” Proudhon

The notion of God played very little part in James’s analysis of religion. The idea of an “external inventor” handing out favours in exchange for prayers, sacrifices or good behaviour was too literal and mean-spirited to have any bearing on the “hush” of religion as James understood it, and in any case magical thought was losing traction as scientific culture spread through the world. Modern atheists might be fixated on God, but believers were no longer very interested: if they used the word at all, they treated it as an arbitrary name for a “supreme reality” engaged, as James put it, in “a wholesale, not a retail business”, or an authority whose demands on us were only “reinforcements of our demands on ourselves”.

Religion was moving away from supernaturalism, in James’s opinion, and from metaphysics and theology too. The secular “religion of humanity” did not have much appeal for him, nor the communistic “religion of socialism”, but he approved of the “ethical societies” and their “churches without God”, and he warmed to the idea – advocated by his friend Thomas Davidson – of a “religion of democracy”. Democracy, for James and many other Americans, was much more than a constitutional mechanism for selecting politicians: it meant an unconditional love of ordinary humanity, and a willingness to entrust the things we prize to their choices – in short, a quasi-religious renunciation of the will to power. “Religion for religion, the religion of democracy is the one which I think makes to me the strongest appeal,” James wrote: “there is no other, in human affairs, to follow.”

There is something magnificent about a conception of religion that is broad enough to include, say, John Stuart Mill or George Eliot, or even Shelley, Darwin or Karl Marx. But the breadth of James’s definition has its disadvantages too. If those of us who think of ourselves as atheists, rationalists, humanists or secularists are to be classified as religious in spite of ourselves – believers, perhaps, but in a post-theistic style – then we risk entering a Hegelian night in which all cows are black. And there is a danger of forgetting, as James tended to forget, the courage of those sceptics who have laboured over many generations to free us from the enchantments of religion. If there are several different ways of coming to religion, there are several ways of moving away from it too, and an adequate inquiry into religion will need to cover not only religious experience, but irreligious experience as well. George Eliot

The most important force pushing people away from religion has always, I suspect, been what you might call the problem of scale. The Copernican revolution in astronomy – the celebrated transition from “closed world” to “infinite universe”, and the demotion of the earth from a commanding position at the centre of the cosmos to a supporting role circling one of the less distinguished of millions of stars – dealt a prodigious blow to human self-esteem. But even without the benefit of modern cosmology, our earliest ancestors must have been able to sense the paltriness of their hopes and fears compared with the colossal indifference of everything else. Most of us, in the course of growing up, will have been transfixed by the thought that we ourselves, together with parents and all the other figures who stride like giants through our lives, are of very little interest to the rest of the human race, and of no consequence at all to the ambient natural world. I remember, as a devout schoolboy, being halted in mid-prayer by the thought of my minuteness: God in his greatness was not going to spare a thought for little me or anyone I knew, and was probably bored to tears by the whole human fandangle.

Disappointment with religion’s false reassurances about the cosmic significance of our existence is likely to be followed by disillusion with the idea of the soul, considered as a glassy essence residing in each of us and giving us a chance of cheating death and possibly winning eternal life. Nietzsche

The 18th-century theologian Emanuel Swedenborg may have comforted a few followers with tales of trips to the realm of the immortals and accounts of domestic routines in charming heavenly cottages. But anyone with a bit of imagination will soon have realised that such arrangements might prove quite inconvenient. Apart from the impersonal vastness of a heavenly city big enough to hold us all, there would be the embarrassment of overlapping personal relationships. Someone who has got through several partners on earth might like the idea of living with all of them in heaven, but they might take a very different view, even if they had no partners of their own to take into account. In the abstract, many of us might like the idea of being reunited with our parents, but we would shrink from the prospect of living with them continuously, world without end, not to mention their parents and their parents’ parents all the way back to the common ancestors of us all.

Apart from these logistical difficulties, the idea of a soul surviving into an afterlife is conceptually vulnerable too: even if an immaterial soul could somehow inhabit our bodies during our lifetimes, it could hardly retain its individual personality once it had shed its mortal coil. John Locke grappled with the issue when he tried to explain “personal identity” in terms of memory, and James himself – inventor of the “stream of consciousness” – concluded that the soul was no more than a “hot place” in a “succession of fields of consciousness” or a “habitual centre” of “personal energy” – not the sort of thing, in short, that could keep itself together without its anchorage in the social relations of a living human body.

The only intelligible hope for individual survival of death involves the physical resurrection of the body, but in spite of familiar images of corpses bursting out of their graves at the sound of the last trump, the idea is too weird to be taken seriously, especially when you consider the problem of reuniting scattered body parts or reversing the effects of fire, flood, erosion and decay. The only remaining possibility is survival at the cost of individuality – an option, traditionally associated with Buddhism, which was embraced with stoic cheer by Spinoza, before achieving some popularity in the 19th century. Ralph Waldo Emerson, for example, suggested that we approach immortality as our consciousness expands to “the full circle of the universe” and eventually merges with the “one mind” that is active everywhere – “in each ray of the star, in each wavelet of the pool.” A moving thought, no doubt, except that an afterlife without individuality seems more like a euphemism for death than a triumph over it. JS Mill

Apart from the problem of scale and the problem of the soul, the other main source of religious disenchantment is the problem of morality. As little children we may have been open to the idea that the moral code we are brought up to is underwritten by religion, but as soon as we notice that different groups have different moralities, just as they have different languages and meal-times and dress-codes, the idea starts coming under strain. And even if it could be proved that one particular version of morality bears the divine seal of approval, we might still choose not to adopt it: we might prefer to follow John Stuart Mill, who argued that it would be best for everyone if each of us would choose our own “plan of life,” rather than suppressing our idiosyncrasies or allowing “the world” – including religious institutions – to make our choices for us. Even if we could prove the existence of a god offering us unambiguous moral advice, the idea that we ought to accept it unquestioningly was, in Mill’s opinion, simply “the most morally pernicious doctrine now current”.

He reserved a special contempt for humanistic secularists who imagined they could extract a kernel of moral truth from the shell of religious superstition – separating the practical teachings of Jesus, for example, from his pretensions to divine authority. “Mankind have, as a race, hitherto grounded their morality mainly on religion,” Mill wrote; “and if their religion is false it would be very extraordinary that their morality should be true.” And Christian morality, in Mill’s view, was most definitely false, because it tried to replace noble ideals of generosity and magnanimity with a system of “self-interested inducement”, an “essentially selfish” incentive scheme promising paybacks in a future life for any losses incurred here on earth. Far from being supported by religion, therefore, morality was liable to be corrupted by it.

Each of these motives for irreligion – problems of scale, of the afterlife, and of morality – makes the idea of God less comforting than it would otherwise be; but none of them constitutes an argument for atheism. Believers of a post-superstitious persuasion – followers of Kierkegaard for example – might indeed see them as hymns to divine glory: paeans to god not as a miraculous personal trainer or jealous cosmic controller, but as what you might call a memento absurdi, a guardian of fragility, contingency, mystery and incommensurability, and a reminder that however clever you may be, there will always be an awful lot of things you do not understand. William James

Opponents of religion – anti-clericals, humanists, rationalists or whatever we want to call ourselves – ought to recognise that religion is a complicated box of tricks, containing much wisdom as well as folly, along with diversity, dynamism and disagreement. And we need to realise that many modern believers have moved a long way from the positions of their predecessors: as Mill once said, they may believe they are loyal to an old-time religion when in reality they have subjected it to “modifications amounting to an essential change of its character”. In particular, they may not accept the idea of God as an actually existing entity, so arguments for atheism will not disturb them; and they will be aware that there has always been more to religion than belief in God. The dividing lines between religiosity and secularism, or between belief and disenchantment, are not getting any clearer as time goes by, and if there has been a lot of traffic travelling from the camp of religion to the camp of disbelief in the past couple of centuries, it has followed many different paths, and is bound for many different destinations.

Illustrations by Scott Garrett