AC Grayling and John Gray are two of Britain’s best-known public philosophers, consistent and relentless in arguing for their respective world views. There, however, the similarities end. Grayling is the approachable, upbeat carrier of the Enlightenment torch, while Gray is the gloomy critic of uplifting myths of human progress and rationality.

If you think that the heavier the world weighs on the thinker, the more heavyweight the thought, then it is obvious which of the two new books by this contrasting duo you’d reach for first. Gray’s The Soul of the Marionette sees him bringing human beings down a peg or two once again, although by now he’s done it so often that we’re virtually subterranean already. This time, his target is our illusions of freedom.

The territory may be familiar for Gray, but he has plotted an original and interesting route around it, introducing us along the way to the writings of relatively unknown figures, such as the 17th-century clergyman Joseph Glanvill, 19th-century Italian poet Giacomo Leopardi, and the Russian science-fiction writers Arkady and Boris Strugatsky. Gray avoids the predictable, standard sceptical argument that human beings are puppets who can’t see their own strings, arguing instead that it is because we are not mere marionettes that we lack freedom. We are cursed with a self-consciousness that makes us want to fight our fates and achieve a state of being that allows us to defy our limitations. The modern way of trying to achieve this is by knowledge. We think the truth will make us free, but the more we find out about ourselves, the more we realise that we are the products of unconscious forces and biological impulses over which we have no control.

Gray advises that we stop fighting this. If we can do this, then “rather than trying to impose sense on your life, you will be content to let meaning come and go”. Paradoxically, it is only by accepting how little freedom we have that we can live with the only kind of freedom which that is possible, the freedom simply to live in accordance with our natures. Once humans stop “looking to ascend into the heavens, they can find freedom in falling to Earth”. This turn inwards is a kind of godless religiosity in which we accept our fallen state.

For many, this sounds like a counsel of despair, in which we give up trying to improve the world or fight our human frailty. Gray invites this charge of pessimism by not stressing clearly that it is possible both to seek improvement and expect that this can only be done partially and incrementally.

Still, Gray is hardly an optimist, unlike Grayling, who is the kind of enthusiastic rationalist Gray likes to scorn. But Grayling’s enthusiasm and conviction should not be confused with naive whiggishness. He does believe that through science and learning we have made progress, but he does not think the march forwards is inevitable or that our gains are irreversible. One recurring theme of The Challenge of Things is precisely that we take our freedoms for granted and that the price we must pay for them is eternal vigilance.

For the most part, however, the problem with his book is not its arguments, which are overwhelmingly sensible, articulate and eloquently expressed. It is rather that it is a somewhat motley collection of short pieces, many of which make for perfectly good journalism but whichthat don’t merit reprinting. Some commentaries on current affairs, for instance, seem too of their moment. Many started life as book reviews, and while it’s fun to see Paul Johnson’s book on Socratescorrect being skewered, if it really has as little merit as Grayling claims, it doesn’t deserve an afterlife being discussed between hard covers.

Grayling does not think that humanity's march forwards is inevitable

AC Grayling’s book is best approached as philosophical Jaffa Cakes. Photograph: Alamy

The book is best approached as a kind of packet of philosophical Jaffa Cakes, to be popped a few at a time to get the brain energised. In contrast, Gray’s often bitter fare is sometimes half-baked or over-egged, but it provides much more to chew over and digest.

Both writers are at their best when presenting their positive theses, but neither can resist taking pot shots at old enemies. Gray is particularly keen to point out how “secular thinking is shaped by forgotten or repressed religion”, but since even Richard Dawkins cheerfully accepts that his cultural heritage is Christian, I can’t believe many atheists would deny this. Grayling also has a tendency to insert dismissive parenthetical comments, such as to ask whether “faith-led institutions of higher education” is an oxymoron, or to wonder whether “postmodern thought” is a “contradiction”. This kind of knowing derision might be a crowd-pleaser at a humanist gathering, but in print it comes over as cheap, arrogant and simplistic.

In one essay, Grayling describes public intellectuals as people who are “acquainted with both history and the history of ideas, who can take from them insights of relevance to the present, and who can effectively communicate new ideas and insights as a result”. How much is new in both these books, especially Grayling’s, is moot. But in every other respect, both Gray and Grayling have demonstrated once again that they fulfil the role of public intellectual admirably well. We are fortunate to have them both, to see both sides of their argument about the sources of human hope presented with intelligence and vigour.

The Soul of the Marionette is published by Allen Lane (£17.99). Click here to buy it for £14.39. The Challenge of Things is published by Bloomsbury, £16.99. Click here to buy it for £13.59