When a party fares as badly as Labour did this election, there are so many reasons for defeat that everyone can simply pick their favourite. Jeremy Corbyn, Brexit, an overstuffed manifesto, an inept campaign, hostile media, Conservative lies.

All of these and more are held up by competing factions. But perhaps the most influential account of Labour’s failure is the one coming from commentators who insist the result vindicates the argument they have been making for a decade: that the left has deserted its “traditional” working-class base by becoming too liberal. Labour was correct, holds this view, to move left on economics but it should seek to win back socially conservative voters by “moving right on culture” – or, to use the phrase of the economist Paul Collier, learn to “talk the language of belonging”.

This is not just an issue for the UK. In western Europe, the established social democratic parties have all seen a long-term decline in support among parts of their working-class base, linked to changes in capitalist production and the rise of political movements that focus on identity and values. The beneficiaries have often been rightwing populists, part of a global wave of movements whose solution to the turbulence of capitalism is a retrenchment of borders, be it physical or cultural.

These movements might differ in style and approach but they all propose to redefine their respective nations along narrower lines: against cosmopolitan elites, against religious, cultural or racial minorities, against outsiders. To regain lost ground, goes the argument, the left must make a fuller embrace of nationalism – something it has failed to do in the past because the intellectuals who shape party policy are squeamish about it.

What the 'language of belonging' almost always seems to boil down to is tougher immigration control

What the advocates of this position usually fail to mention is that centre-left parties have repeatedly done this – and fallen flat on their faces. The link between Labour and voters usually described as its “traditional” working-class base has been decaying since the early 2000s, and subsequent leaders have tried a variety of techniques to win them back. Under Tony Blair – a politician adept at using the language and symbols of the nation (his ascent to power in 1997 was accompanied by a photo of him posing on the white cliffs of Dover) – Labour tried to shore up its nationalist credentials with highly visible crackdowns on asylum seekers and “foreign criminals”, as the BNP began to appear in former industrial towns once dominated by the left.

As prime minister, Gordon Brown noisily promised “British jobs for British workers” in the wake of the financial crash but came across as insincere – a feeling crystallised by the moment he was caught on mic calling a member of the public bigoted.

His successor as Labour leader, Ed Miliband, made a more considered attempt to marry a slight shift to the left on economics with the conservative communitarian ideas of Blue Labour: the result was that he managed both to lose Labour’s strongholds in Scotland and be portrayed by the London media as un-English (in some quarters with a nasty dig at his Jewish refugee heritage).

There is nothing wrong with calls for the left to talk about belonging. What matters is how belonging is defined – and who gets to do the talking. All too often the assumption is that the left should drape itself in patriotic clothing as a shortcut to power. This ends up with the worst of both worlds: the politicians look insincere, and they reinforce the notion that belonging can only be articulated in exclusionary ways.

The conversations we need to have cannot be conducted via a media and political establishment that excludes much of the country it purports to speak for. Those who push for the left to become more nationalistic are overwhelmingly part of the liberal elite they criticise. They are Oxford economists, magazine editors, politics professors and professional thinktankers. What they propose is to add a dose of nationalism to a political system where power remains in the hands of small circles of technocrats. We will hear a lot of talk in the coming months about the need for Labour to listen to voters. Good. But what use is listening if you are then planning to offer yet more of the same? How about finding ways to help people speak – and take decisions – for themselves?

Furthermore, what “the language of belonging” almost always seems to boil down to is tougher immigration control. Yet it’s essential to remember that this is a policy, not a campaign slogan. As the hostile environment and the Windrush scandal show all too clearly, this isn’t just talk. Hardening immigration controls comes with a body count – whether or not the affected individuals are regarded as “belonging” to the nation.

Immigration control can also work to entrench division rather than make people feel more secure. Think of the UK’s asylum policy, a flashpoint for rightwing discontent for several decades. In response to a xenophobic moral panic fuelled by rightwing newspapers in the late 1990s, New Labour placed asylum seekers on a cashless vouchers system and forced them to live in “dispersed” accommodation around the UK, often in deprived areas. The Cameron government privatised asylum seekers’ housing in 2012, pushing people to smaller, more peripheral towns.

All this was done under the guise of making British citizens feel safer and more secure. In practice it treats refugees like unwanted guests and gives the impression to longstanding communities that they are having problems “dumped” on them. This is the kind of detail that the people pontificating about “belonging” rarely grapple with.

If Labour were to make a nationalist turn now, it would risk finishing off the parts of its base it has so far retained. Cas Mudde, a leading expert on the far right, argues that copying the right is a dead end and that while centre-left parties may be in trouble, the values that underpin them still have widespread support. But that doesn’t mean the left can’t talk about the nation: the different, contradictory things it means to people, what this says about our histories and where we would like to be in the future.

That won’t be comfortable. But talking about belonging – about solidarity, community, looking out for one another – is precisely where the left should be strongest. The question has to be how people divided by class, region, history, experience and outlook can build something better together – not which of those groups should be prized over the others, and treated as more authentic.

• Daniel Trilling is author of Lights in the Distance: Exile and Refuge at the Borders of Europe