Francis Fukuyama’s sense of grievance, expressed in the preface to his new book, is patent. Ever since he published his essay “The End of History” in 1989, he writes, he has been asked whether this or that event didn’t invalidate his thesis. The event could be “be a coup in Peru, war in the Balkans, the September 11 attacks, the global financial crisis, or, most recently, Donald Trump’s election and the wave of populist nationalism”:

Most of these criticisms were based on a simple misunderstanding. I was using the word history in the Hegelian-Marxist sense – that is, the long-term evolutionary story of human institutions that could alternatively be labelled development or modernization. The word end was meant not in the sense of termination, but “target” or “objective”. Karl Marx had suggested that the end of history would be a communist utopia, and I was simply suggesting that Hegel’s version, where development resulted in a liberal state linked to a market economy, was the more plausible outcome.

It must be frustrating to spend nearly 30 years battling vainly against what you consider a simple misunderstanding of the idea at the heart of your work. But if Fukuyama’s idea of the end of history has not been understood, one reason may be that it is incoherent. He tells us that it is not a terminus but instead an objective. At the same time he says history is the evolutionary story of human institutions. As understood by Darwin, however, evolution has no objective. The chief achievement of the theory of natural selection is to expel teleology from biology and explain the development of life without reference to objectives.

The inexorable implication is that if history is an evolutionary process, it has no objective either. Fukuyama’s end of history remains what it has always been, a farrago of Hegelian metaphysics and ersatz evolutionary theory.

Even if we allow Fukuyama his confused concept, it is unclear why anyone should accept his particular version of it. Is “a liberal state linked to a market economy” the only possible alternative to communism? If the world’s economies converge on any single system – an unlikely prospect – might it not be something like Chinese state capitalism or the oligarchical capitalism that currently prevails in the US? Might not the political systems that prevail be mostly tyrannies and illiberal democracies?

Continuing his complaint, Fukuyama writes: “My critics missed another point. They did not note that the original essay had a question mark at the end of the title.” Like the Fabian socialists Sidney and Beatrice Webb, who dropped the question mark in later editions of their eulogy to Stalinism Soviet Communism: A New Civilisation? (1935), Fukuyama eliminated the interrogative in the book that followed his original essay. Is he now reinserting it?

It would seem not. In Identity, he still posits that liberal democracy is what in his original essay he called “the final form of human government”. But now he recognises the possibility of liberal democracies “decaying or going backwards” and reiterates a claim made in his earlier work: that liberal democracies have not solved the problem of thymos. Thymos, Fukuyama explains, is the part of the soul that “craves recognition of dignity”; isothymia is “the demand to be respected on an equal basis with other people”; and megalothymia is “the desire to be recognised as superior”:

Modern liberal democracies promise and largely deliver a minimal degree of equal respect, embodied in individual rights, the rule of law and the franchise. What this does not guarantee is that people in a democracy will be equally respected in practice, particularly members of groups with a history of marginalisation. Entire countries can feel disrespected, which has powered aggressive nationalism, as can religious believers who feel their faith is denigrated. Isothymia will therefore continue to drive demands for equal recognition, which are unlikely to ever be completely fulfilled.

There is a certain banality in this analysis, which recurs throughout the book. Near the beginning, he writes: “Unlike thymos, which is a permanent part of human nature… the modern concept of identity emerged only as societies started to modernise a few hundred years ago.” Later, he observes: “There is nothing wrong with identity politics as such… It becomes a problem only when identity is interpreted or asserted in certain ways.” In the book’s final sentences he summarises his argument: “Identity can be used to divide, but it can and has also been used to integrate. That in the end will be the remedy for the populist politics of the present.” The politics of identity, then, can be good or bad. “In the end”, however, it will prove to be good.

It is not a terribly illuminating conclusion. Why is the politics of identity so strong at the present time? Fukuyama says very little on this question. The book’s 14 chapters consist of potted intellectual history interspersed with thumbnail sketches of recent political events. Illustrating the apparently neglected truth that “human psychology is much more complex than the rather simple-minded economic model suggests”, he takes the reader through Plato and Aristotle, Luther, Hegel and Rousseau, among others, in the search for what he calls “a better theory of the human soul”. Along the way, he offers an account of modern politics as featuring two competing demands for recognition and dignity, one being concerned with respect for peoples, the other focusing on universal human rights.

Though there is nothing novel in this story it contains some useful insights. Fukuyama is perceptive on the rise of national identities. He notes that according to the British-Czech anthropologist and sociologist Ernest Gellner (1925-1995) the nation state was not, as many liberals assert today, a reversion to tribalism. Instead it helped make possible modern economies based on exchanges between strangers and thereby marked a step away from tribal communities. Of course there have always been elements of ethnicity in nation states. But rather than national identities being reversions to a pre-modern past, they are an integral part of the modern world.

Where Fukuyama falls down is in having no credible account of the rise of identity politics. “After 1989”, he tells us, countries from the former Soviet bloc “threw off communism and rushed into the EU, but many of their citizens did not embrace the positive liberal values embodied in the new Europe”. They did not “make an effort to entrench liberal values in their citizens” and were “among the least diverse societies in the developed world”. In other words, former Soviet bloc countries failed to reach the level of development of the rest of Europe.

This hardly explains the advance of the far right in Italy or in Sweden – currently one of the most diverse societies in the developed world and one in which progressive versions of liberal values have been inculcated for generations. Nor does underdevelopment in the former GDR explain the popularity of Alternative für Deutschland throughout the rest of Germany. What is missing in Fukuyama is any recognition of the part played by the progressive consensus in fuelling the forces that are now overthrowing it. Might not rising illiberalism be a reaction against the liberal extremism that is expressed in projects such as a Europe without internal borders?

According to Fukuyama, liberal societies may have been remiss in not taking thymos seriously enough. As part of their enthusiastic embrace of globalisation, they may have allowed inequality to get out of hand. Otherwise, there is nothing much wrong with them. Kwame Anthony Appiah also seems to find little in liberal societies to explain the popular revolt against their ruling elites. The problem seems to be in the people themselves, many of whom are gripped by an erroneous concept of identity, a “conceptual mistake” that – as he puts it in The Lies That Bind – “underwrites moral ones”. If this intellectual error could be corrected, ethnic nationalism could be left behind and liberal cosmopolitanism salvaged.

In five of the book’s six chapters, Appiah considers an idea of collective identity that developed in the 19th century which he believes needs re-examining in the 21st. The first and most successful such chapter concerns religion, which Appiah rightly argues is not centrally about belief but about practices and communities. In later chapters he examines theories in which identities are grounded in nations, races, classes and cultures. Writing with graceful informality and often citing his own experiences, he argues that all of them embody a single error – that of “supposing that at the core of each identity there is some deep similarity that binds people of that identity together. Not true, I say; not true over and over again.”

The error that Appiah denounces is “essentialism”: the theory that members of the same group share some inner essence that explains what they have in common. We are plural all the way down; no identity defines us entirely, or is more fundamental than any of the rest. It is the false doctrine of essentialism, he seems to believe, that underlies the many conflicts in which humans attack and kill fellow human beings on account of the identities they ascribe to themselves and others. If this fallacy could be eliminated, these savage conflicts could be avoided or moderated. The remedy for populism, then, is the anti-essentialist philosophy of nominalism, according to which ideas that split up the world into definite kinds of thing are simply tools we use to simplify our dealings with an environment actually composed of an infinity of particulars. If we appreciated the limitations of our general concepts we would understand that singular human identities are illusions, and stop fighting over them so fiercely.

As a nominalist myself, I am sympathetic to Appiah’s claim that unambiguous identities are illusions. But when they are widely accepted, illusions become social facts – and they can be very powerful. He tells us that some of the 20th century’s worst crimes “were perpetrated in the name of one people against another with the aim of securing a homogeneous nation”. No doubt this is so, but it is enormously oversimplified.

Consider movements demanding secession. Secessionists are not necessarily possessed by ideas of national homogeneity, though some have been. More often, they fear they will be losers in a state where the majority belong in a different community. It is not only their identity they fear will be lost. The material conditions of their lives – housing, land ownership, access to resources and services – will also be threatened. Such fears helped break up Yugoslavia after the death of Tito, and destroyed several post-colonial African states.

Appiah consistently underrates the power of these material factors. Focusing on the complexities of social hierarchy and downplaying the role of structural inequalities in power, his chapter on class is the weakest in the book. Like many other liberals today, he would benefit from injecting a dose of Marxian materialism into his thinking. Societies are not made up of only the concepts and beliefs humans form about them. They are partly composed of the physical resources human beings control, which in turn help shape the ideas that people accept.

Appiah’s analysis of nationalism is also weak. Are nationalist movements really the result of people being swept off their feet by the musings of a disparate bunch of 19th-century European thinkers? Gellner’s theory that the nation state is a functional prerequisite of a modern economy may be far from a complete explanation, but it is more credible than the fantastical notion that nationalism is a mere misunderstanding.

Appiah announces at the start that he “won’t offer an explanation of why identity talk has exploded in my lifetime – a fascinating question, but one for intellectual and social historians”. But if anti-essentialism can’t help explain the present, what is the point of writing a book about it now? It is not as if refuting essentialism will remedy the evils of identity politics. The far right parties that are advancing in one country after another won’t be stopped by deconstructing the concept of the nation. Populist nationalism is a symptom, not a cause of the disorder of liberal societies.

Like Fukuyama, Appiah has no doubt that he can describe a plausible liberal future. Near the end of the book he writes that once we abandon organicism – the view of cultures as organic wholes that forms one version of essentialism – “we can take up the more cosmopolitan picture in which every element of culture – from philosophy or cuisine to the style of bodily movement – is separable from all the others”. As instances of this cosmopolitan way of life, he writes movingly of the poet Cavafy’s Alexandria and the city of Trieste. His examples are telling, for they point to an awkward fact.

Each of these great cosmopolitan cities was a modern enclave of empire. Founded in conquest, the structures of which they were part were exploitative and discriminatory; but they enabled diverse communities to co-exist and interact peaceably, and when they collapsed the result was ethnic cleansing on a large scale and much more homogeneous societies than had existed before. Appiah might have cited London, today the supremely cosmopolitan city. But London is a creation of the multinational British state, itself an relic of monarchy and empire.

At this point an inconvenient question suggests itself. What if plural identities survive and thrive best not in modern nation states but in some of the antique institutions that preceded them? How curious if a cosmopolitan civilisation – Appiah’s 21st-century ideal and Fukuyama’s end of history – should turn out to be in the past.

John Gray is the New Statesman’s lead book reviewer. He appears at the Cambridge Literary Festival, in conversation with Rowan Williams, on 24 November: cambridgeliteraryfestival.com

Identity: Contemporary Identity Politics and the Struggle for Recognition

Francis Fukuyama

Profile, 240pp, £16.99

The Lies that Bind: Rethinking Identity

Kwame Anthony Appiah

Profile, 272pp, £14.99