Why would anyone not vote Labour? If person A knocks on your door and offers you £1,000, no questions asked, and person B offers £100, why refuse A? The question has long puzzled experts. Labour’s Jeremy Corbyn has pledged a great deal more in public spending than the (relatively) parsimonious Tories. Yet the Tories are running at 11 points ahead in the polls and appear within reach of victory. They have come first in seven of the last 10 elections, and according to at least one poll now hold a majority over Labour even among working-class voters.

In the early days of mass democracy it was thought populist politicians would bankrupt nations by promising to borrow and splurge ever more of “other people’s money”. Then the opposite theory held sway, that as the poor got richer they would vote Conservative. By the 1960s, Mark Abrams and Richard Rose famously asked Must Labour Lose? The answer was yes, unless the party changed.

Under Harold Wilson and Tony Blair, Labour largely aped the Tories and won elections. It reassured the wavering centre of its un-radicalism. But the conservative undercurrent soon dragged voters back to the right, a tendency recently exacerbated by the titanic struggle over Brexit. It has left political analysts largely baffled.

The clearest answer came from the US academic Jonathan Haidt, in an attempt to understand why on earth poor people voted for George W Bush. Why would they want a wider wealth gap, worse healthcare and more corporate greed? Haidt’s answer was that these “left behinds” saw politics as about more than money. It was about their families and collective safety, about local and group identity, about faith and a morality seen as personal, not ordained by the state. Yes, they cared about other people and fairness, but they also cared about themselves.

Haidt’s thesis was deeply troubling for Democrats – as it should be for Labour in Britain. Analysis by Matthew Goodwin and Roger Eatwell into who voted for Brexit in Britain and Donald Trump in the US found the tribes remarkably similar, with support in both cases more widely spread across ages and classes than initially supposed. Indeed, average leave and Trump voters had above-average incomes. What they shared was a concern about the continuity of their identity and way of life. They feared outsiders and newcomers. Brexit was cheered as much in struggling ex-industrial towns as in affluent Conservative counties.

‘Under Harold Wilson and Tony Blair, Labour largely aped the Tories and won elections.’ Wilson on the steps of No 10 in April 1966. Photograph: Fox Photos/Getty Images

Most explanations have suffered from a failure to separate liberal contempt from dispassionate analysis. The decline of party politics in favour of group identity is often attributed – at least by Londoners – to issues of ethnicity and religion. Provincial Britain sees it differently. Its group identity is that of neighbourhood and town, assailed by outsiders of all sorts, with children seduced by bright lights, and replaced with migrants and interminable executive estates.

As the political scientist Eric Kaufmann points out, income and social background are having ever less bearing on voting behaviour: “Small-c conservative working-class voters have migrated to the Conservative party because of immigration and Brexit. On the other side, successful educated cosmopolitans opt for Labour or the Lib Dems.” This is most vividly seen in the correlation between remain voting and higher education. This is not a matter of economics, he says, but “a measure of psychological openness and a liberal worldview”.

What may be deplorable must not become incomprehensible. I find leave voters resent what they see as their natural concern for the integrity of their community – or even the sovereignty of the British state – being dismissed as xenophobic or racist. When instructed to “change or die”, they reply that we are all dead one day. Yes, a dynamic economy must be open to change – including economic migration – but only if the beneficiaries can persuade the sufferers that it is in their interest. Either way, it will depend on trust. Nothing so destroyed faith in David Cameron’s government as its disregard of localism in planning and public services. People who do not trust those ordering them to change will seek revenge – even if it costs them money.

Brexit was, of course, change – but to some voters it was an imagined reversion to a more familiar past. It must be the most truly conservative vote in modern British history. It is why Boris Johnson was wise to stick with it, though unwise now to tear up treaty trading relations with the EU. It is also why Corbyn’s reversion to the Labour party of the 1940s – to a land of nationalisation, council estates and workers’ rights – may be shrewdly conservative rather than plain daft. It should reassure old Labour, like Tories preaching a return to empire.

I have always thought it dangerous for liberals to assume that conservatives know in their hearts that conservative values are selfish and shallow. Humans are by their nature conservative. The new populism gets this, that politics is no longer “the economy, stupid”. It is about things people find more important than money. Those who condemn Brexit fail to see it was never really to do with the EU, any more than it concerned the gross national product. It was about how far Britons can define and protect the character of where they live.

Just now, the Conservative instinct understands this. So, in his Brexit hesitancy, does Corbyn. With Brexit over, it will be liberals who have a mountain to climb.

• Simon Jenkins is a Guardian columnist