In the last 10 years or so, the rise of American evangelicalism and the menace of Islamist fundamentalism, along with developments in physics and in theories of evolution and cosmogony, have encouraged a certain style of aggressive, often strident atheistic critique. Books such as Richard Dawkins's The God Delusion and Christopher Hitchens's God Is Not Great have sold in the millions. Beyond the unlikely success of these books, there has also been the spread of atheist and secularist websites and blogs, some of them intellectually respectable, others more dogmatic and limited (ie, pretty atrocious). The events of 11 September 2001 were the obvious spur. In The End of Faith, the American writer Sam Harris argued that as long as America remains swamped in Christian thinking, it will never defeat militant Islamism, since one backward religious system cannot prevail over another backward religious system. Atheism would be the key to unlock this uneasy stalemate. Academics such as Dawkins and Daniel Dennett have broader projects, perhaps – for them, the removal of our religious blinkers will result in a proper appreciation of the natural world, and of science's ability to describe and decode it.

I can't be the only reader who finds himself in broad agreement with the conclusions of the New Atheists, while disliking some of the ways they reach them. For these writers, and many others, "religion" always seems to mean either fundamentalist Islam or American evangelical Christianity. Hinduism, Buddhism, Judaism and the more relaxed or progressive versions of Christianity are not in their argumentative sights. Along with this curious parochialism about the varieties of religious belief comes a simplistic reading of how people actually hold those beliefs. Terry Eagleton and others have rightly argued that, for millions of people, religious "belief" is not a matter of just totting up stable, creedal propositions ("I believe that Jesus is the son of God", "I believe that I will go to heaven when I die", and so on), but a matter of more unconscious, daily practice ("Now it is time to kneel down, face Mecca and pray"). This kind of defence of the deep embeddedness of religious practice has been influenced by Wittgenstein – for whom, say, kissing an icon was a bit like loving one's mother; something that cannot be subjected to an outsider's rational critique. Wittgenstein was obviously right, though this appeal to practice over proposition can also become a rather lazy way, for people like the Catholic Eagleton, of defending orthodox beliefs via the back door – as if a bishop encouraged his flock by saying, in effect: "It doesn't matter what you believe. Religion is not about propositions, but about practices. So stick at those practices: just keep on doing the church flowers and turning up every Sunday."

We know that plenty of people hold religious beliefs that are also propositions – they stand up and recite creeds on Fridays, Saturdays and Sundays; they can tell you who will be punished in hell, and how; they believe that Allah is the one God, and so on. Prayer itself is a proposition: it proposes that God exists, and can be communicated with. Rather than simply declaring all religious belief to be non-propositional, which is manifestly untrue, it would be more interesting to examine what might be called the practice of propositional beliefs. We know that people believe all kinds of things, as propositions. But how do they believe them? In this area, the New Atheism has nothing very interesting to say, except to wish away all such beliefs.

But people's beliefs are often fluctuating and changing – it is why people lose their faith, or convert to faith in God. If you spend any time asking people what they believe, how they believe, and why they believe the propositions they espouse in church or temple or mosque, you find that there is nothing very straightforward about propositional belief. Recently, I spent some time with two Christian believers, both ordained. One is an academic theologian and university chaplain, the other a religious affairs journalist. The academic theologian was walking with me in a university town, and began a sentence, "I believe." And then he caught himself, and added: "I don't know what I believe, at the moment." A few weeks later, I met the religious affairs journalist, who had for several years been a parish priest. During the course of our conversation, he asserted: "It is impossible to be a serious Christian and believe in heaven and hell." When I, who was raised in a strongly and conventionally religious home, expressed surprise and suggested that once one stops believing in heaven one might as well stop believing in God, he said, more vehemently: "It's exactly the opposite: not believing in heaven and hell is a prerequisite for serious Christian belief." Trapped in the childhood literalism of my background, I had not entertained the possibility of Christian belief separated from the great lure and threat of heaven and hell.

The New Atheism is locked into a similar kind of literalism. It parasitically lives off its enemy. Just as evangelical Christianity is characterised by scriptural literalism and an uncomplicated belief in a "personal God", so the New Atheism often seems engaged only in doing battle with scriptural literalism; but the only way to combat such literalism is with rival literalism. The God of the New Atheism and the God of religious fundamentalism turn out to be remarkably similar entities. This God, the God worth fighting against, is the God we grew up with as children (and soon grew out of, or stopped believing in): this God created the world, controls our destinies, sits up somewhere in heaven, loves us, sometimes punishes us, and is ready to intervene to perform miracles. He promises goodies in heaven for the devout, and horrors for the damned. Since militant atheism interprets religious faith, again on the evangelical or Islamist model, as blind – a blind leap of faith that hurls the believer into an infinite idiocy – so no understanding or even interest can be extended to why or how people believe the religious narratives they follow, and how often those narratives are invaded by doubt, reversal, interruption and banality.

There is a telling moment in The God Delusion when Dawkins speculates on why countless generations of people believed in God. How could belief in an illusion have persisted for so long? Dawkins suggests that we have evolved an HADD, a "hyperactive agent detection device": "we hyperactively detect agents where there are none, and this makes us suspect malice or benignity where, in fact, nature is only indifferent." His example of this elementary mistake comes from the episode of Fawlty Towers in which John Cleese's car breaks down. Cleese gets out and starts hitting the car. This is an example of HADD, and by extension, of mankind's belief in God. Now, do you really think that offering a minute from Fawlty Towers is an adequate analogy for millennia of religious belief? This is not about whether one believes in God or not. One can be an unbeliever and find this a bit feeble. Marx said that the study of religion was the most serious project an intellectual could have. If I told you that the history of warfare, say, could be "explained" by some recent discovery of a particular receptor in the brain, that Agincourt and Austerlitz, Antietam and the Ardennes were all essentially the same thing, because produced by a universal delusion, what would I have told you about the nature of warfare, of politics, of statecraft, of the enormous mass mobilisations that Tolstoy characterised as "the swarm-like life of mankind"?

One good place to study that "swarm-like life", and to see religious belief seriously represented and seriously examined, is the modern novel – from, say, Melville and Flaubert in the 1850s to the present day. Melville, Dostoevsky, George Eliot, Jens Peter Jacobsen, Tolstoy, Virginia Woolf, Beckett, Camus – and in our own time José Saramago, Marilynne Robinson and JM Coetzee – have all shown sustained interest in questions of belief and unbelief; many of them have struggled with the departure of God. Because they are novelists, they want to see both sides of a theological argument, and so they can't afford to do what militant atheism does, which is merely caricature any form of belief it doesn't approve of. They offer narratives of belief, and novelistic narratives make real the ambiguity, the contradiction, the intermittence, even the absurdity and comic irrationality of our intellectual lives. In a beautiful passage in Moby-Dick, Melville says that the ocean constantly moves and heaves like a human conscience. That could be said of our mental life, too.

Part of the weakness of current theological warfare is that it is premised on stable, lifelong belief – each side congealed into its rival (but weirdly symmetrical) creeds. Likewise, in contemporary politics, the worst crime you can apparently commit is to change your mind. Yet people's beliefs are often not stable, and are fluctuating. We are all flip-floppers. Our "ideas" may be rather as Woolf imagined consciousness, a flicker of different and self-annulling impressions and convictions. What if you were a strong Christian believer, and you woke one night, terrified by the sudden awareness that God does not exist? Hours pass in this unillusioned crisis, and then blessed sleep finally returns. The next day, you wake up and the awful doubt – a thing of the night – has mysteriously disappeared. You continue to "believe in God". But what does such belief now mean? If it has not been annulled by the doubt of the night, does it now contain the memory of its inversion, as a room might trap a bad smell?

An essay or work of polemic finds it hard to describe the texture of such fluctuation, whereas the novelist understands that to tell a story is to novelise an idea, to dramatise it. There is no need to make a tidy solution of belief; to the novelist, a messy error might be much more interesting. The Brothers Karamazov offers a famous example from the 19th century – a novel in which the author, a fiercely Christian believer, argued against his own beliefs so powerfully that many readers are swayed by Ivan Karamazov's atheism (as Dostoevsky feared might happen). For a contemporary instance, there is the recent work of Coetzee, who has explored the contradictory and irrational ways in which people hold ideas and propositions. In Disgrace, Elizabeth Costello, and Diary of a Bad Year, essentially religious feelings (such as atonement, shame, self-mortification) rub up alongside apparently more rational and propositional beliefs (for example, Elizabeth Costello's belief that eating animals is the moral equivalent of the Holocaust). This mixture of the sentimental and the intellectual makes for an unstable compound, and Coetzee wants to dramatise, I think, how ideas are not just held but actually lived – which is to say, how they are often lived irrationally. When a college president asks Elizabeth Costello if her vegetarianism comes out of her moral conviction, she evades apparently rational argumentation, and replies, religiously, that it "comes out of a desire to save my soul".

Polemicists want to prosecute intellectual contradictions, novelists to explore them. In the work of the wonderful 19th-century Danish atheist novelist Jens Peter Jacobsen (1847-1885), we find a writer doing the same contradictory dance as Dostoevsky, but the other way round. Jacobsen was a passionate atheist and early translator of Darwin. Where Dostoevsky argued against his own Christian beliefs, Jacobsen seems to argue against his own atheism. The hero of Jacobsen's great novel Niels Lyhne is a convinced atheist, but he gives far greater credit to God's presence than he can ever give to God's supposed non-existence. At a crucial moment near the end of the book, when his small son is dying in his arms, Niels breaks down and prays to God, even as he judges the moral lapse from his proud atheism that such weak-minded prayer represents. Like many atheists, Niels seems unable to stop invoking a God whose existence he is supposed not to credit. Niels is always in a relation with God, even when he declares his non-belief in that God; Jacobsen's novel brilliantly dramatises how Niels seems only ever able to banish God, not kill him off.

Contemporary atheistic and theological polemic tends to assume that we all simply choose our beliefs – and can thus choose not to have any belief. That may be true of privileged intellectuals, but there are surely many millions who don't feel they have the freedom to choose belief or unbelief; instead, their beliefs choose them. Woolf seems to understand this in To the Lighthouse, when she has Mrs Ramsay, who thinks of herself as an unbeliever, suddenly express conventional Christian belief. In section one of that novel, Mrs Ramsay is sitting looking out of the window at the lighthouse, and thinking of many things at once (of children, of marriage, of her husband, of how the greenhouse will cost £50 to mend, and so on). Suddenly, a conventional phrase of religious solace floats into her head: "We are in the hands of the Lord." Straight away, because she does not believe in God, she repudiates the belief: "But instantly she was annoyed with herself for saying that. Who had said it? Not she; she had been trapped into saying something she did not mean." She is irritated with herself for giving expression to something she does not believe. But Woolf's question remains: who had said it? If Mrs Ramsay didn't say it (or didn't mean it), then who did? Woolf's novel is famous for being full of scraps of Victorian verse, and is rich in biblical language, and so one could say that "We are in the hands of the Lord" is nothing more than a Victorian relic, an old phrase that Mrs Ramsay remembers from her childhood. But one of the novel's central questions turns on what it means to continue to need or make use of a religious language whose content is no longer believed in. If Mrs Ramsay doesn't believe the words, why did they slide into her consciousness? Mrs Ramsay does not speak the words; they speak her. And perhaps the answer to the question "who had said it?" is: "God." Or: "God, for a minute." Or: "God, interrupted."

There is an amusing clip on YouTube, in which Dawkins confronts Rowan Williams. Dawkins asks the archbishop of Canterbury if he really believes in miracles such as the virgin birth and the resurrection, happenings in which the laws of physics and biology are suspended. Well, not literally, says Williams. But, says Dawkins, pouncing, surely Williams believes that these are not just metaphors? No, says the archbishop, they are not just metaphors, they are openings in history, "spaces" when history opens up to its own depths, and something like what we call a "miracle" might occur. Dawkins rightly says that this sounds very nice but is surely nothing more than poetic language. Williams rather shamefacedly agrees. The scene is amusing because both men are so obviously arguing past each other, and are so obviously arguing about language and the role of metaphor. Dawkins comes off as the victor, because he has the easier task, and holds the literalist high ground: either the resurrection happened or it didn't; either these words mean something or they do not. Williams seems awkwardly trapped between a need to turn his words into metaphor and a desire to retain some element of literal content.

Both men could find themselves in Moby-Dick. For in that novel, Melville explores precisely the question that hovers over the Dawkins-Williams exchange. Can God be literally described, or are we condemned to hurl millions of metaphoric approximations at him, in an attempt to describe him? After all, in Melville's novel, the white whale is symbolic of both the devil and of God, and the writer tries very hard to describe the nature and mass and temperament of that indescribable whale: Melville uses scores of different metaphors to capture the essence of the beast, and fails. It cannot be captured in words. Only when the beast is killed will it be captured. Melville's novel is a kind of ironic counterpart to Aquinas's belief that God can only be described by what he is not. Melville, who fluctuated violently between belief and unbelief, seems to have been terrified by the idea that if God cannot be reached by metaphor, then God is only a metaphor.

Dawkins is dead to metaphor, and tries to annul it by insisting on the literal occurrence, contained in actual words, of the virgin birth and the resurrection. And Williams insists that such literalism misses the target, and instead has recourse to the metaphor of "event", of a "space" opening up in history, an indefinably miraculous aberration. One feels sympathy for both sides – and perhaps simultaneously a plague on both their houses – because Dawkins seems so bullishly literal, and Williams so softly evasive. Contra Dawkins, God should be allowed some metaphorical space; but contra Williams, God's presence in the world, God's intervention, should not surely be only metaphorical. God is not just a metaphor.

Of course, Melville does not suggest any solution to the Dawkins-Williams argument (not least because there couldn't be one); as a novelist, his task is, as Chekhov once put it, just "to ask the right questions". Which is not a small thing.

This article is adapted from a recent Weidenfeld lecture at St Anne's College, Oxford.