“It is not prejudiced to worry about immigration,” Ed Miliband suggests. Possibly. But no one is worried about immigration. Yes, 77% of people in the UK want immigration reduced, and 56% want it “reduced a lot” – but this is not worry or concern, rather a hardening anti-immigrant consensus. As to whether this is prejudiced, the evidence is clear: polling shows that most people have a negative opinion of immigration, despite them not having any negative experience of it. In fact, people in areas with most immigrants are least likely to express anti-immigrant sentiment.

Popular hostility toward immigrants is determined by the perceived big picture, which polling data also shows most people get badly wrong. These errors are not neutral. The fact that people greatly overestimate the proportion of immigrants who are asylum seekers, for instance, matters largely because of the culture of suspicion and disbelief about refugees.

It is hard to see how this could not be prejudiced.

Among defenders of immigration, whether liberal or left, there is a reluctance to confront the prospect of mass, popular racism. For example, John Harris is not untypical in arguing that anxiety over immigration is not racist but a response to desperate day to day experience produced by mass migration. Experience has little to do with it. Most Britons have been hostile to immigration for decades, long before the waves of EU migration in the last decade. What has changed now is the political salience of this hostility.

Another tendency is to acknowledge racism only to treat it as the result of elite or rightwing misdirection exploiting people’s real concerns. If this was the case, it would be enough to expose “Ukip lies” as Nick Clegg attempted to do in debating Farage. But he was mauled. That’s because the dodgy “facts” were mere supports for a wider morality fable according to which Brits are cheated by mass immigration. People believe the fable, so they accept the facts.

People aren’t merely dupes. They take an active role in the construction of their beliefs. They are invested in them, finding enjoyment and solace in them. To explain popular racism we have to stop explaining it away, and look at the conditions that make sense of it.

The dominant sentiment of this racism is resentment. People are convinced that immigrants have taken something from them. Social resentment of this kind is integral to the competitive ethos of neoliberalism: given a vicious struggle for scarce resources, there is tremendous paranoia about “undeserving” people getting things unfairly.

Moreover, the development of neoliberalism in the UK is inextricable from the politics of Britishness. Thatcher shutting off immigration was as central to her agenda for national renewal as a strong pound or Falklands fever. New Labour’s competition state, as part of its Britishness agenda, embarked on an offensive against failed multiculturalism and attacked non-EU immigration. Immigrants have been continually identified as undeserving leeches, a security menace, an existential threat to British values – and thus a fitting target of resentment.

Not everyone is equally susceptible to this kind of resentment. Research suggests that those most prone to a resentful nationalism are those sectors of the working and middle classes who have been on a downward trajectory for decades. Their Britain, as they see it, no longer exists. And they link the class injuries inflicted by de-industrialisation, underinvestment and precarity to the decline of Britain’s global position in the postcolonial world, and the loss of their old sense of racial and national superiority.

The credit crunch and ensuing crises serve as a punctuating moment in the development of this ideology. The bankers were almost universally reviled, but rightwing voters also blamed Labour for spending too much money on politically correct groups like immigrants. As they see it, the productive majority are being sapped by parasites at the top and at the bottom.

In this context, the fantasy of a restored Britishness offers consolation. The assertion of belligerent anti-immigrant ideas, meanwhile, appears as a defiant reclamation of lost enjoyment in the face of tyrannical “political correctness”.

To suggest that most people may be racist about immigration is not a counsel of despair. There are differing levels of commitment to racist beliefs. For example, there is evidence that a margin of people who are hostile to immigration would be much less so if they were apprised of the facts. Further, of the remainder who would remain anti-immigrant, those who would put it ahead of everything else in order to vote for a party like Ukip are a minority.

There is also an anti-racist minority, comprising young people, trades unionists and those who are the target of racism themselves. Their mobilisation in support of immigrants would make a big difference to the national conversation, divide the anti-immigrant bloc and help hold back the racist tide.

However, this is purely defensive, and short term. The racist backlash has been constructed over a long period, and it has been entwined with experiences that are integral to the neoliberal model of growth now supported by all dominant parties. To undo it will take time and the development of an alternative, together with a language with which to express social grievances and channel resentment where it belongs.