In Uncle Tom’s Cabin, there is a moment when Augustine St Clare, who owns Tom, suggests that he is better off as a slave than he would be as a free man. “No,” insists Tom. “Why Tom?” asks St Clare. “You couldn’t possibly have earned, by your work, such clothes and such living as I have given you.” “Know’s all that Mas’r,” says Tom. “But I’d rather have poor clothes, poor house, poor everything and have ’em mine, than have the best, and have ’em any man else’s.”

It has long been a challenge, particularly for those who are comfortable, to understand why anybody who is struggling would choose to be worse off. Liberals can take it particularly personally when those who would most benefit materially from a change in policy or circumstance opt to reject it. The assumption is that they must be misinformed, ill-informed, uninformed, stupid, naive or cruelly misled.

Whether they are poor Americans demonstrating against healthcare provision they don’t have, or direct beneficiaries of European Union funding voting to leave the EU, those who act “against” their material interests invite a mixture of befuddlement and derision.

Since becoming poorer is not a course of action any poor person would consciously take, goes the argument, they are clearly not acting rationally and deserve what is coming.

Quite why well-off liberals in particular would find someone voting against their material interests such a baffling idea is odd. They do it all the time. Whenever they vote Labour, or for any party that plans to raise taxes on the wealthy and redistribute income, they vote to make themselves worse off materially. True, they are better positioned to take a hit than those at the other end of the income scale. But there’s more to it than that. They do so, for the most part, because when it comes to politics they don’t just vote for their own financial wellbeing. They are thinking about the kind of country and world they want to live in, and the values that they hold dear.

The notion that working-class voters approach politics differently is extremely patronising. Indeed it is precisely the kind of attitude that provides fodder to the rightwing culture warriors who rail against the “coastal elites” in the US and “do-gooders”. There really are some liberals who think that they know what’s better for working-class people than working-class people themselves do.

This is not just a problem in and of itself – infantilising people in the name of their own advancement is a bad thing – it is counterproductive. When you start from the premise that those who disagree with you are acting illogically or are too unsophisticated to understand their own interests, no meaningful political engagement is possible – that would demand first understanding the logic and then challenging, converting, subverting or otherwise engaging it in the hope that you might change someone’s perspective and win them round.

This has, of course, been a particular challenge when it comes to Brexit, where the two things we know are that poor people will be the most adversely affected by Britain leaving the EU – particularly if there is no deal – and that the poorer you are, the more likely you were to vote for it.

With some notable exceptions, remain advocates have responded to this apparent conundrum by forsaking respectful engagement in favour of a combination of face-palming at the stupidity of lemmings going for a leap and promising Armageddon when they land. This didn’t work in the run-up to the referendum. And it’s not working now.

This is partly because leave voters don’t believe the hype. A recent poll showed a significant majority of them believed Brexit posed a less serious crisis than either the financial crash or the miners’ strike. In other words, it’s not that they don’t understand things could get worse; they just feel they’ve been through worse. That may, as Fintan O’Toole argued in this newspaper recently, reflect the complacency of those who have only known stability. “Only a country that does not really know what the collapse of political authority looks like would play this game,” he wrote. We won’t know until it’s too late.

But also many did not vote purely for their material interest, but for something bigger that they thought more important. Polling by the Centre for Social Investigation revealed that remain voters significantly underestimated the importance that leave voters attached to sovereignty. The UK making its own rules came a close second out of four (immigration was first) in the reasons why people voted leave. When remain voters were asked why they thought people had voted leave they put the UK making its own laws last, after “teach British politicians a lesson”.

Embodied in that preoccupation with sovereignty, I believe, was a notion of what this country has been, has become and might be – a story many British, and particularly English, people tell themselves about a once independent and impregnable distinct island that has lost its autonomy to a faceless potage of bureaucrats from Babel and how this is a chance to break free. That story did not come from nowhere. From the Falklands war to Fritz, the New Labour bulldog, the entire political class has colluded in its construction. It has now been leveraged by opportunists and is consuming the political class whole.

It is a story enduring enough that when someone argues, “If you do this your factory might close,” you might respond: “It’s not my factory and ‘they’ve’ been closing factories around here for years. But it is my country and I don’t want ‘them’ messing with it.” In this story “they” is a moving target. It could be immigrants, it could be Brussels, it could be foreign companies. The only thing “we” know for sure is it’s not “us”.

I think that story’s deeply flawed. It is mythical about the past: those who evoke the wars conveniently forget that they could not have been won without allies. Those who evoke the empire conveniently forget that it could not have been maintained without brutality (as the current furore over Winston Churchill’s legacy illustrates). It is obtuse about the present: countries evolve, borders shift, identities develop. Our royal family is German; our favourite food is Indian; and if people could only agree on how to spell Muhammad it would be England’s most popular name for a boy. We are not who we were; nor should we seek to be. And it is fanciful about our future: our political sovereignty, like everybody else’s, is primarily constrained not by Brussels, but international capital. And since you can’t vote to leave that, staying in or out of the EU will make us less effective, but not more independent.

But I also think, in the absence of other stories, it is compelling. Far more compelling than the threat of a scarcity of fresh vegetables and a run on the pound if we crash out with no deal. Compelling enough that some would suffer to see it through. Our challenge is not to mock, but to tell a better story. One that includes them, has a future for all of us and, ultimately, turns “them” and “us” into “we”.

• Gary Younge is a Guardian columnist