In The Reckless Mind (2001), his previous study of the role of the intellectual in politics, Mark Lilla identified “a new social type” which he saw as having come into being along with the rise of those two great malign systems of the 20th century, fascism and communism. This type he named the “philotyrannical intellectual”, a term as unlovely as the species it identifies. Now, 15 years later, the world is swarming with philotyrannical activists and thinkers, in Asia and the Middle East, in the former Soviet Union, and in the US – anywhere, in fact, in which new messiahs of various hues have stepped forward to promise a future return to a past paradise.

That earlier book dealt with the likes of Martin Heidegger, Carl Schmitt, Alexander Kojève, and those irrepressible French twins, Michel Foucault and Jacques Derrida. In a superb brief afterword, entitled “The Lure of Syracuse”, Lilla went back all the way to Plato and his ill-judged attempt to turn the tyrant Dionysius of Syracuse into a philosopher-king, in order to trace the lineal descent of latter-day apologists, some deluded, some unashamedly committed, for Hitler, Stalin, Mao, Osama bin Laden and so on.

Lilla, one of our most incisive public intellectuals, has now produced another volume of essays, on what Lionel Trilling called, somewhat grandiloquently, the “bloody crossroads” where literature and politics meet. It is a timely and illuminating study of political reaction, historical and contemporary, and its devastating effects on the present-day world and, most likely, the world of the foreseeable future as well.

However, the publication of the book itself has been overshadowed by the controversy provoked by an article Lilla wrote in the New York Times last month, following Donald’s Trump’s election, which argued that in recent years “American liberalism has slipped into a kind of moral panic about racial, gender and sexual identity that has distorted liberalism’s message and prevented it from becoming a unifying force capable of governing”. This seemed a reasonable observation, especially to those of us who were in the US in the months before and after the election and witnessed, and felt, the levels of rage and resentment among many millions of voters – Hillary Clinton’s “deplorables” – who felt themselves excluded by the liberal, well-heeled and, it should be emphasised, multiracial middle class, and plumped for Trump whether they liked him or not.

Lilla’s critics have accused him of urging a return to the bad old Mad Men days when western democracies were dominated by white middle-class males, and racism, sexism and general intolerance were the rule of the day, such evils being the necessary price of national unity, in the US at least. However, as anyone will realise who read the piece carefully, this is not what Lilla thinks, and is not what he was recommending. At its simplest level, the article merely observed that to emphasise and celebrate diversity to a level that damages social cohesion is not only foolhardy, but dangerous, and would end in disaster for liberalism – the disaster philosopher Richard Rorty predicted a decade ago, and which has now come to pass.

Lilla’s critics would do well to read The Shipwrecked Mind, for its range of reference, its common sense, its subtlety and persuasiveness. The tenor of some of the reactions to his New York Times article portrayed him as a reactionary; this book is an ideal corrective to such absurd charges. It is divided into three sections, Thinkers, Currents and Events – Lilla happened to be living and working in Paris last year at the time of the attack on the Charlie Hebdo offices. An introduction gives the book its title, and an afterword, “The Knight and the Caliph”, begins with an anecdote about Don Quixote and goes on to consider not only the horrors but the peculiarities of militant Islamism, such as its radical nostalgic bent: “For the apocalyptic imagination, the present, not the past, is a foreign country.”

The introduction, which shows Lilla at his scintillating best, points to the fact that today’s university libraries will offer the reader hundreds of books in numerous languages on the topic of revolution, whereas “on the idea of reaction you will be hard put to find a dozen”. This is an odd state of affairs, since it is not any revolutionary force that has created the blood-boltered world of today, but the force of reaction, whether in Putin’s Russia, in Trump’s would-be-great-again America, or in the festering deserts of Iraq and Syria.

“Reactionaries”, Lilla gently points out, “are not conservatives.” This is frequently overlooked, or forgotten, or not conceived as a possibility in the first place, by middle-ground liberals. But the reactionary is every bit as fierce in his convictions, and every bit as determined that his views shall prevail, as the most uncompromising Jacobin. However, there is this difference: that the reactionary looks not to the future but backwards towards the past. “Where others see the river of time flowing as it always has, the reactionary sees the debris of paradise drifting past his eyes,” Lilla writes. In a fine phrase, among many fine phrases, he identifies “the militancy of his nostalgia” as the mark of the true-born reactionary, which is what makes him “a distinctly modern figure, not a traditional one”.

Subtlety and persuasiveness … Mark Lilla

Among the three Germans Lilla examines in his Thinkers section is Leo Strauss, who specialised in the political philosophy of the classical era, and who after emigrating to the US taught for many years at the University of Chicago, where he was a strong influence on reactionary intellectuals such as Allan Bloom and Paul Wolfowitz, the latter a prominent neocon, former president of the World Bank and deputy secretary of defense under George W Bush. An admirer of Heidegger, Strauss in his work turned back to the early days of European civilisation, identifying an originary tension between the mainly Jewish, mystico-religious search for salvation, and the Greek assertion of the cognitive and shaping powers of human reason. This tension was at the heart of the Enlightenment, which sought, Lilla writes, “to create a new kind of society that would be free of both religion and classical philosophy – of Athens and Jerusalem”. Strauss chose Athens: like the reactionary Heidegger before him, he considered that after the Greeks, humanity entered on a slow plunge into inauthenticity and bad faith.

One of the main strands of Lilla’s argument is that what the rest of us might see as progress, reactionaries regard as the signs of widescale retrograde betrayal. “Every major social transformation leaves behind a fresh Eden that can serve as the object of somebody’s nostalgia.” And nostalgia, here, can be far more potent than, for instance, hope: “Hopes can be disappointed. Nostalgia is irrefutable.” In our contemporary predicament, following the EU referendum, Trump and now the defeat of Italy’s reformist prime minister Matteo Renzi, what countless voters think is hope for a return to better times is in fact nostalgia for a time that never existed.

Like John Gray, Lilla sees clear to the heart of modern-day millenarianism and finds there the old, old story of longing for a lost golden age and the expectation of a brave new world to come. He recalls the immense influence Oswald Spengler’s The Decline and Fall of the West had on philosophers and politicians after the first world war, but he also finds strong traces of “declinist” thinking in modern-day self-styled revolutionaries such as “apocalyptic deep ecologists, anti-globalists and anti-growth activists”. Nor is it just on the left that Spengler’s legacy is still alive: it is also there “in the writings of radical political Islamists, whose story of the secular west’s decline into decadence, and the inevitable triumph of a vigorous, renewed religion, has European fingerprints all over it”.

In the section Currents, Lilla considers various manifestations of reaction, such as what we might call the “neo-Catholic” Brad Gregory’s The Unintended Reformation, the pretensions and conclusions of which he delicately fillets, and the strange, ongoing rehabilitation of St Paul among thinkers on the far left and the far right, including Hitler’s jurist Schmitt on one side, and the French Maoist Alain Badiou on the other. Badiou, Lilla writes with fine irony, is one among a number of victims of “a very old political romanticism that longs … to break free and feel the hot pulse of passion, to upset the petty laws and conventions that crush the human spirit and pay the rent”. However, their patron saint, he insists, is not St Paul, but Emma Bovary, who read too many romances, and dreamed too many deluding dreams.

• The Shipwrecked Mind is published by NYRB. To order a copy for £8.19 (RRP £9.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.