Modern atheism is mostly a continuation of monotheism by other means, and nowhere more so than in radical political movements that claim to have rejected religion entirely. The seminal work on faith-based politics is Norman Cohn’s The Pursuit of the Millennium (1957), where he shows how the patterns of thinking of late medieval millenarians, who believed a new world was coming into being as the result of divine intervention in which the old one was destroyed, have been replicated by modern secular revolutionaries. In communism the agent of this transformation was the human species and in nazism the leader of a “superior race”. But the faith in a redeeming catastrophe was the same.

Arthur Koestler’s Darkness at Noon (1940) is the story of a communist who is swept up in the purges. Arrested and interrogated for crimes against the revolution he did not commit, he ends by confessing to them and being executed. The novel is a study in the ruthless logic of faith, which – whether transcendental or secular – demands human sacrifices as the price of salvation. Koestler renounced his own faith in communism in 1938, partly as the result of a mystical experience he had while awaiting execution after being captured by Francoist forces while working as a Comintern agent in Spain. He was freed in a prisoner swap, but his life has changed for ever.

When Prophecy Fails (1956) by Leon Festinger, Henry Riecken and Stanley Schacter is a study of an American UFO cult that expected the imminent end of the world. Infiltrating the group, Festinger and co found that when the cataclysm failed to occur the cultists did not lose their faith – on the contrary, it was strengthened. When belief systems are contradicted by facts, the beliefs are rarely renounced. More often, they are reinterpreted and thereby reinforced. Humans are more interested in preserving an internally coherent worldview than in testing their view of things against events. Nearly always, faith trumps facts.

The best book ever written by a philosopher on religion is William James’s The Varieties of Religious Experience (1902). Approaching faith in a pragmatic and empirical spirit, James – brother of the novelist Henry James – focuses on experiences rather than beliefs. Faith is a matter of how one lives, not a theory that can be verified or falsified. His book was much admired by Ludwig Wittgenstein, a deeply religious person who never subscribed to any religious belief.

Friedrich Nietzsche’s The Gay Science (1882) was, among other things, an attempt to fashion an atheism that broke not only with religious belief but more importantly with the ways of thinking that monotheism had inculcated. For Nietzsche, modern atheism had been a byproduct of Christianity. Rightly, to my mind, he believed a genuinely free-thinking atheism would begin by questioning the surrogate faiths – in “humanity”, science, progress and so on – that replaced Christianity among those who liked to think they had rejected religion. Modern atheism was not, as its adherents imagined, an alternative to faith, but a way of closing the mind to doubt.

Sadly, but perhaps predictably (he was after all the son of a pastor), Nietzsche went on to produce an ersatz faith of his own in his myth of the Superman, a Christ-like figure that redeems humankind from nihilism – the condition of meaninglessness that supposedly befalls us when we no longer have any idea of God. The free-thinking atheism he originally envisioned remains elusive.

• John Gray is the author of Seven Types of Atheism (Penguin, £9.99). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99.