Knocking on doors for Labour for more than 100 hours in London, Bedford and Milton Keynes showed me the stark difference in voters’ attitudes by age. Among a section of older, white voters in both more and less affluent areas, I saw a visceral hatred of Jeremy Corbyn, and sometimes Diane Abbott. How did the demonisation of Corbyn and the Labour party under his leadership – as documented by media analysis conducted at the LSE and Loughborough University – impact so strongly with them in 2019 but not in 2017?

Although on the face of it that demonisation has been raw and relentless, it actually only circled around a key charge, rarely making it explicit, so it has taken four years for low-engagement voters to absorb it fully.

The key charge against Corbyn is that he fundamentally believes British lives are of equal value to the lives of others. His opponents wouldn’t put it so bluntly, but this is what it has always been about. Hence the series of confected outrages – from not bowing deeply enough at the Cenotaph to ruling out pushing the nuclear button – that built a treasonous charge sheet as absurd as it was banal.

Smears such as Corbyn ‘supporting' the IRA gained more traction as time went on

It seemed impossible to defend Corbyn against this unspoken indictment. Smears such as Corbyn “siding with Putin” over the Salisbury poisoning, when caution about trusting the judgment of British intelligence agencies was cast as support for the Russian version of events, or “supporting” the IRA, gained more traction as time went on.

Even Corbyn’s commendable record of campaigning against the geopolitical grain, such as for dispossessed Tamils, Chagossians and Palestinians, came to be seen as evidence that he didn’t know which side he was supposed to be on. A symbolic moment of the campaign was the first leaders’ debate, when Corbyn highlighted the impact that the climate crisis would have on the poorest people in the world and a section of the audience responded with groans and someone shouted, “Here we go again!”

When people talk about having paid into the system all their lives, as I heard repeatedly at the doorstep, they’re not just talking about national insurance payments and the benefits they’re entitled to. They’re talking about loyalty to a state they expected to be their exclusive patron – and they saw a Labour leader who seemed to invite the whole world to his allotment, offering homemade jam to all, no matter which flags their ancestors spilt their blood for.

With such voters, retired or coming towards the end of their careers, Corbyn’s collectivist language of what we could build together left them sceptical and uncomprehending. It seemed more zero sum to them, where one person’s gain must be another’s loss. A small hoard has been salvaged from the UK’s long post-imperial decline, and only those whose fealty is proven can claim their share.

This isn’t about a chauvinistic sense of racial or national superiority. I encountered no Brexit optimism, no sense of “Believe in Britain” boosterism. On the contrary, people were fixated on the inevitability of scarcity, and the need to guard against naive hope.

The feeling that Corbyn’s loyalties were too wide for a national leader is a materialist concern, because people were worried that he would be profligate and squander the security inherited from times when Britain was more powerful. That’s why the leadership was not trusted to deliver the popular policies in the manifesto. Sensible investments such as state-provided broadband came to be seen as giveaways. Taxing the rich was unpersuasive, as many people just thought it impossible. These voters wanted the patronage of the powerful, not to challenge their power.

The good news is that people are not crying out for more racism or war. Immigration itself didn’t come up once in all my conversations, and indeed polls show declining hostility to it. Corbyn’s opposition to military adventurism is popular, although his pacifist principles are not.

Pandering to so-called legitimate concerns will fail, as it always does for Labour, but so will labelling large chunks of the electorate racist. Despite ethical injunctions to “call out” prejudice, there are no electoral prizes to be won from naming racism as such, except in the most egregious cases. We need to reduce the salience of nativist and nationalist identity. Let’s start by abstaining from the self-indulgence of “gammon” jokes and other temptations to sharpen cultural and intergenerational divides.

I also canvassed many young, working-class people who were not engaged with politics. Many had never heard about class politics at all, and expressed confusion and boredom regarding Brexit. The idea of voting for a party to tax the rich to pay for redistribution and public services was completely novel, and generally immediately attractive. It was amazing to see how quickly and instinctively they grasped a leftwing agenda while saying they had never thought about it before.

There is a huge opportunity for the left to make inroads with younger non-graduates in towns, but imaginative strategies are needed to reach them, since sending enthusiastic activists out from the cities for a few weeks before an election is too little too late, especially when younger voters are rarely at home to answer the door.

Labour still runs local governments across the country. This needs to be seen as an opportunity for popular participation – engagement in civic and political life on a local scale – to combat the feeling of powerlessness that saps the ability of people to imagine a radically different future. With five years in the wilderness at the national level, Labour may have to explore the potential of municipal socialism, as pioneered in Preston and seen in Barcelona.

• Luke Pagarani is a trustee of the Human Capability Foundation charity