“The starting point is always numbers,” the late, great theorist Stuart Hall wrote about reporting on migration and race in 1979. But as “soon as you say numbers, it doesn’t matter how you wrap it up – there is only one lesson to be drawn, the numbers are growing. There are too many of them”. This week Michael Gove demonstrated that this is as true now as it was 40 years ago, when he issued a warning about immigration levels. The government’s net migration target may be no more, but the same old anti-immigration arguments are still treated, in some quarters, as gospel.

On Monday morning, Gove was in the pages of the Times weighing in on the supposed “dangers” of “uncontrolled” and “unlimited” immigration. Although Labour has yet to agree its migration policy for the election, Gove referred to the motion at Labour conference that called on the party to maintain and extend free movement, denouncing this position as “extreme”. It would, he wrote, “mean massive pressure on public services – creating a shortage of school places, putting a huge strain on the NHS and increasing demand for housing”.

Three days earlier, the Conservatives had announced a supposedly liberal policy – the introduction of “half-price” visas and a fast-track process for medical professionals coming to the UK. Yet the Tories who want doctors and nurses to come from all over the world to work in the NHS consider most other migrants a threat to that same health service. When it suits them, they celebrate the important “contributions” that immigrants make to the UK. The rest of the time, however, we’re told that immigrants are “overwhelming” schools, housing and the NHS – that they are “drains” on the same public services the Tories have spent years underfunding.

We might expect little else from the party of “go home” vans and the “hostile environment” policy. But Gove gives voice to some of the myths firmly established in the debate – some of which have been challenged by Labour, though not always robustly enough. They’re not as prevalent or seen to be as important as they once were, for all kind of reasons – including the belief that immigration will fall if we leave the EU.

But the myths are still there and the Conservatives are eager to perpetuate them – despite Windrush, despite the people who risk or lose their lives trying to cross borders and despite the reams of evidence showing how the UK’s immigration system destroys people’s lives. If those 39 people had been found alive instead of dead in the back of that lorry in Essex, we wouldn’t have heard expressions of shock and sorrow from ministers – we’d have had calls for their deportation and promises to “crack down” on “illegal immigrants”.

This is where the numbers come in. For decades, there have been attacks on too much immigration of a “certain kind” (not everyone who moves to the UK and settles here for at least 12 months is considered an immigrant – this is very much a racialised term). It has been treated as bad for some fabled British “culture”, as well as public services and wages. And perhaps most importantly, for politicians seeking to justify their regressive policies and rhetoric, it’s taken as given that the public don’t like “high levels” of immigration.

Following this well-worn script, Gove argued that any attempt to keep and extend free movement would make Labour “out of touch with the British people”. And when shadow cabinet members have refused to be drawn out on arbitrary numbers, they have been told that people need to know if immigration will go up or down.

Conventional thinking is that people’s “concerns” about immigration should be accepted not only because they are “legitimate”, but because they are a natural response to “too much change”. “Controls”, politicians have argued for decades, are necessary for “good race relations”.

But people are not made racist or anti-immigrant by immigration. Public opinion is produced in a system that has stigmatised particular groups of immigrants for decades, and given meaning to those migration numbers. This is fuelled by politicians such as Gove weaponising immigration and claiming they speak for the people, even as we’re told immigration is no longer a major public “concern”.

On the left, at least, the point of mapping public opinion is not only to reflect it – in turn entrenching the views that people already hold – but to understand it so that you can, when necessary, try to change it. In this election, that means not indulging policies that would let migrants into the country and then expel them depending on how they “contribute” to the economy. It means not repeating one of the great anti-immigration myths, the claim that immigration “undercuts” wages – no matter how certain parts of the labour movement might frame it. And it means vocally defending immigration and seeing that as integral to the struggle for a more equal world.

People have always moved and they will continue to move. The Conservatives might talk about controls and limits in this election but the problem is not and has never been immigration – it’s anti-immigration politics.

• Maya Goodfellow is the author of Hostile Environment: How Immigrants Became Scapegoats