Like new Labour, so-called New Atheism did not just replace the old variety but, for a while at least, almost totally occluded it. Atheism is now sometimes discussed as though it began with the publication of Richard Dawkins's The God Delusion in 2006.

To put these recent debates – or more often than not, flaming rows – in some sort of perspective, a thorough history of atheism is long overdue. The godless may not at first be pleased to discover that the person who has stepped up to the plate to write it comes from the ranks of the opposition. But Nick Spencer, research director of the Christian thinktank Theos, is the kind of intelligent, thoughtful, sympathetic critic that atheists need, if only to remind them that belief in God does not necessarily require a loss of all reason.

Spencer's story is designed to illuminate our present, so he understandably restricts himself to western Europe from the late middle ages onwards. It is a compendious though not definitive account, which shows why atheism is not simply the natural result of the rise of scientific knowledge, and religion a simplistic vestige of more ignorant times. Spencer rightly points out that, far from being enemies of religion, science and rationality were often most enthusiastically championed by men and women of faith. Locke and Newton were, for instance, both profoundly motivated by their Christianity.

In the long run, however, the church is being slowly undermined by the critical powers of inquiry it helped unleash. As Spencer himself argues, a "fateful shift" occurred in the 17th century when rationalists such as Descartes and the Cambridge Platonist Henry More sought to justify Christianity with reason. The idea was that atheism would be "defeated on the battleground of its own choosing", but once the fight moved there, religion found itself permanently on the defensive, on a long-term retreat despite the odd counterattack.

Much of the narrative is strictly historical, but there is also a polemical edge. Spencer wants his history to support three contentions, two of which should not be contentious at all. That we should talk about "atheisms rather than atheism" is self-evident. While the likes of Saint-Simon and Comte had a naive faith in the power of science and reason to create an orderly, happy utopia, later existentialist thinkers such as Nietzsche saw that "much must collapse because it was built on this faith" and looked forward only to a "long dense succession of demolition, destruction, downfall, upheaval".

Nor is there much to disagree with in the claim that atheism was from the start "a constructive and creative phenomenon", not just concerned to tear down the old order but to erect something more enlightened and rational in its place. Even the various atheistic libertines who thought all morality was an illusion believed that a world without constraint would be superior to the religious status quo.

What is more debatable is the contention that "the history of atheism is best seen as a series of disagreements about authority" rather than one primarily about the existence of God. "To deny God was not simply to deny God," writes Spencer. "It was to deny the emperor or the king who ruled you, the social structures that ordered your life, the ethical ties that regulated it, the hopes it inspired and the judgment that reassured it."

This is certainly true. But it does not follow that the tussle between religion and atheism is political rather than philosophical. Take his discussion of the early reformation in the 16th century. "Hundreds of Christians wrote thousands of pages demolishing the theological presuppositions of their opponents," he rightly says, before adding, "the fact that those theological differences might be a cipher for political or social threats is a nuance easily lost amid the aroma of cooking flesh."

Of course, there were political and social factors involved in the various disputes and schisms. But to conclude that therefore their theological contents were irrelevant "ciphers" is a jump too far. It is a false choice to say that the battles must "really" be either political or metaphysical: the messy reality is that they are jumble of both.

Similarly, Spencer wants to encourage us to see religious teachings as more moral than factual or historical. This view goes back to at least the 16th century when Cardinal Cesare Baronio asserted: "The Bible tells us how to go to heaven rather than how the heavens go." It is a neat aphorism, but of course it makes no sense to be told how to get to heaven unless there is a heaven to get to. Beliefs about what is real and what is not are impossible to expunge from all but the most postmodern of theologies.

Spencer is here promoting the conception of "religiosity as pattern of life rather than a set of verifiable propositions". On this view, what matters is not whether difficult doctrines such as eternal damnation or even Christ's resurrection are true or false, but that a life guided by such ideas is somehow richer, more complete, more directed towards a higher good. If that is right, then atheists who have criticised religion for its doctrines have spectacularly missed the point, "tilting at theological windmills". But as Spencer himself argues, we didn't see "theological liberalism redrawing the lines" until the last decades of the 19th century, and, even then, only a minority accepted the new map.

However, following John Gray, he is right to say that there is something odd about the kind of secular humanism that says all we need to do, to quote the famous bus campaign slogan, is accept "There's probably no God. Now stop worrying and enjoy your life." Believing that human beings are special is natural if you believe God created us apart from other animals, not if you believe we are higher primates whose brains evolved to help us survive and reproduce. This should certainly call into question naive atheist faith in the power of secular reason, even if Spencer goes too far when he suggests it ends up undermining its very basis, "sawing through the branch on which the atheist sat".

Atheists have more grounds to protest that Spencer puts too much weight on some specific periods and episodes that cast them in a bad light. The state-sponsored atheism of the communist world, for example, is discussed at length, even though few western atheists saw the collapse of the Berlin Wall as any kind of defeat for their far more liberal worldview.

He also says "the abrupt death of logical positivism" – the early 20th-century philosophical movement that declared all religious and metaphysical talk as literally meaningless – "marked the end of one of the most significant atheist philosophical traditions". On the contrary, it was a shortlived blip that made the mistake of grossly simplifying the less dogmatic empiricism it had grown out of. The same could be said for the recent "New Atheism spasm".

Although there is plenty here for infidels to argue with, there is much more that is undeniably true and important to know, if you want to understand the complex histories of both present-day religion and atheism. Whether atheism is true, however, depends not on how it got to where it is now, but on how well supported by argument and evidence it now is. History can enrich our understanding of the debate, but it cannot settle it.

• Julian Baggini's most recent book The Virtues of the Table is published by Granta.