Last November, Columbia University historian Mark Lilla published a comment piece in the New York Times, entitled The End of Identity Liberalism. Numbed by Trump’s election victory, Lilla placed the blame largely at the door of “identity politics”, which, he argued, had atomised American politics, undermined civic culture and destroyed the Democrats’ electoral chances. Liberalism, he wrote, “has slipped into a kind of moral panic about racial, gender and sexual identity that has distorted liberalism’s message and prevented it from becoming a unifying force capable of governing”.

Lilla’s essay became the eye of a furious political storm. Some critics suggested that he was whistling in the wind – all politics, they insisted, is necessarily identity politics. Others saw it as an attack on minorities. Katherine Franke, professor of law at Columbia, and a colleague of Lilla’s, claimed that Lilla was doing the “background work of making white supremacy respectable”.

It’s a debate equally significant for politics on this side of the Atlantic. Here, too, the left has considerably weakened, society has become more fragmented and there has developed an equally fraught debate about the politics of identity.

Now Lilla’s op-ed has become a book, The Once and Future Liberal: After Identity Politics. Its publication has reignited the debate over the politics of identity. According to Lilla, the high point of American liberalism came with Roosevelt’s New Deal of the 1930s, which focused not on individual needs but on the collective good. To regain power, Lilla argues, the liberal left needs to rediscover that notion of the common good by adopting a pragmatic form of politics. He is particularly caustic about protest movements such as Black Lives Matter. “We need no more marchers,” he writes. “We need more mayors.”

Between them, Lilla and his critics sum up well the impasse of contemporary politics on the left. Many of his critics cannot see that the politics of identity, far from defending the marginalised and the powerless, fragments the possibilities of meaningful social change. Lilla cannot see that the self-proclaimed “liberal centrist” politics he espouses has helped create the fragmentation of which he despairs. In Europe, too, debates about immigration and multiculturalism, about nationalism and federalism, expose a similar kind of deadlock. The roots of contemporary identity politics lie in the new social movements that emerged in the 1960s to challenge the failure of the left to take seriously the issues of racism, homophobia and women’s rights. The struggle for black rights in America, in particular, was highly influential in providing a template for many other groups to develop concepts of identity and self-organisation. Squeezed between an intensely racist society, on the one hand, and a left often indifferent to their plight, many black activists ceded from civil rights organisations and set up separate black groups.

The universalism that once fuelled radical movements has largely evaporated

This is where Lilla’s celebration of New Deal liberalism looks so threadbare. The “Roosevelt dispensation”, Lilla argues, inaugurated a liberalism filled with “confidence, hope, pride and a spirit of self-sacrifice”. Except that it was not quite like that. It was also a liberalism that accommodated Jim Crow laws, segregation and lynchings. Millions of Americans were excluded from the American “we” that Lilla wants to defend. It was in the struggle against such exclusion that the origins of postwar identity politics lie.

“As identity consciousness has increased among liberals,” Lilla has observed, “political consciousness has decreased.” That is to look at the issue back to front. It is not so much that identity consciousness has diminished political consciousness, but rather that the diminishment of ideological politics has allowed the politics of identity to flourish. In the 1960s, the struggles for black rights and women’s rights and gay rights were closely linked to the wider project of social transformation. But as the labour movement lost influence and radical struggles faltered, from the 1980s on, so the relationship between the promotion of identity rights and broader social change frayed. Eventually, the promotion of identity became an end in itself. The universalism that once fuelled radical movements has largely evaporated.

The erosion of the power of labour movement organisations, the demise of radical social movements, the decline of collectivist ideologies, the expansion of the market into almost every nook and cranny of social life, the fading of institutions, from trade unions to the church, have all helped to create a more fragmented society. These are the changes that have snapped social bonds and hollowed-out civic life.

That hollowing out has been exacerbated by the narrowing of the political sphere, by politics that has self-consciously become less ideological, more technocratic. The Democrats in America have discarded much of their old ideological attachments as well as their links to their old social constituencies. Dick Morris, former chief political adviser to the then president Bill Clinton, whom Lilla lauds, called this the process of “triangulation” – the left stealing the right’s clothes, so that it can appear to be above ideological politics. It was an approach appropriated by Tony Blair for New Labour; many see in Emmanuel Macron’s policies an attempt to fashion a new Gallic version of the same.

It is not, however, through triangulation or managerialism that people bind together. They do so through common struggles for social change. Such struggles enable people to reach out beyond their own identities and give meaning to civic solidarity. It is through such social struggles that we can define what common goals should be, and what we might mean by the common good.

As the influence of the labour movement has declined, and broader social struggles have faded, so “solidarity” has for many become increasingly defined not in political terms – as collective action in pursuit of certain political ideals – but in much narrower terms of group identity. Trump himself is a product of this. So are the anti-immigration populists of Europe. One of the consequences of the mainstreaming of identity politics is that, on both sides of the Atlantic, racism is becoming rebranded as white identity politics.

What Lilla fails to recognise is that the demand for “mayors not marchers” – for pragmatic politics over social movements – is a change that has already happened; and the consequence has been the kind of identity politics he rightly despises. The problem is not that there are marchers rather than mayors. It is, rather, that both marchers and mayors, both activists and politicians, operate in world in which broader visions of social change have faded. How to restore a sense of solidarity based on broader politics rather than narrow identities – that’s the real challenge we face.