Is there a direct line between concern about immigration and what we might call “popular racism”? The idea drives much of the public debate around the subject. But the evidence on public attitudes doesn’t support it.

Earlier this week, Richard Seymour suggested much of the public concern about immigration was the result of prejudice. But he shouldn’t take the toughest headlines as reflecting public attitudes.

As British Future’s pamphlet How to Talk About Immigration, to be published next month, sets out, there is certainly a toxic anti-migrant minority of the public – a sizeable and noisy segment of up to one in four people who hold strong anti-immigration and anti-migrant views. It also finds strong evidence that most of those who are worried about immigration are not anti-migrant. These distinctions should inform effective strategies for challenging and tackling prejudice.

Rob Ford of the University of Manchester, who has led the British Social Attitudes survey research modules on immigration, told us: “There is some racism and prejudice in the immigration debate. But it simply does not fit the evidence to claim that most concern about immigration is rooted in prejudice.”

Most people are not impressed by the record of governments on handling immigration, whether it was the Labour government’s surprise at the levels of European migration after 2004, or the current government’s inability to keep the promises it made before it was elected.

But that is not the same as being anti-migrant. Ask people about the Polish migrants who have come here since 2004 and a clear majority say that they have “made a positive contribution to Britain”, while a surly 20% disagree. That’s a funny kind of xenophobia.

Most people reject hostility to Poles, Bulgarians or Romanians who come here to work and to contribute, while they think there are challenges in the workplace, where nobody gains if migrants are unfairly exploited so that others can be undercut. They still feel there are pressures in communities, in terms of how we handle change and bring people together. That doesn’t make them racists.

Engage with anxieties constructively about how we make migration work fairly, for citizens and migrants alike, and there is a pro-migrant majority to be unlocked.

Those who regard all concern about immigration as racist are also often allergic to even the most inclusive versions of national identity. But with his celebrated opening ceremony to the 2012 Olympics, Danny Boyle spoke for most people, even if even he couldn’t quite reach everyone. While the idea of an inclusive Britishness is generally popular, it has been consistently shown that this is more highly valued by non-white Britons than white Britons. Leftwing white graduates are less keen, but that’s a different story.

What we see is a notable generational shift. Age, rather than class, is now the strongest predictor of attitudes. Younger generations are mostly pretty comfortable with Britain’s diversity. They grew up with it, so it is normal to them.

Their grandparents are considerably more unsettled by the pace with which they have seen Britain change. If you want to understand the populist political phenomenon of Ukip, this generational chasm in attitudes is probably the most important place to start, as the academic studies suggest. It is not a catch-all phenomenon, but a view particularly strongly held by those over 65, who left school at 16, and who live in areas of lower diversity. That group feels “left behind” by how Britain has changed, but most people don’t.

There is clear evidence that there is an anti-racist majority in Britain, not just an embattled “anti-racist minority”.

Of course, it isn’t racist to talk about immigration – as long you do so without being racist.

That’s the debate people want. Anti-racists would have much to gain from that, if they stopped shouting almost as shrilly as the populist xenophobes – which only closes down the conversation that Britain’s moderate majority would like.