It's the influx that never was. New Year's Day, we were told by rightwing politicians and press, would be the day the floodgates opened. Romanians and Bulgarians, free at last to work in Britain without restrictions, would come in their hordes. Beggars and benefit scroungers would be battering on our doors. The country would be swamped.

But when it came to it, there was no sign of them: no special coaches, no temporary camps and plenty of spare seats on flights from Bucharest and Sofia. It's not so hard to work out why. Romanians and Bulgarians have been able to live and work throughout the European Union since 2007. There are already about 150,000 in Britain, 2 million in Italy and Spain, and seven other EU countries lifted working restrictions yesterday, including France and Germany.

No doubt the numbers will pick up, though it won't be on the scale of the east European migration of the past decade. But for months, we have been subjected to a drumbeat of hysteria, as the Tories vied with the nationalist UK Independence party to terrify the public about the coming onslaught and promise ever more meaningless or toxic crackdowns, egged on by a xenophobic media.

Migrants will be charged for emergency hospital treatment at their bedside, the government announced – but that won't apply to EU citizens. The Daily Mail and 90 Conservative activists begged David Cameron to invoke an EU "safeguard clause" to keep the curbs on Bulgarian and Romanian employment in place, while Tory ministers claimed they were being blocked by the Liberal Democrats. It was grandstanding nonsense, as the European commission would have had to agree to it.

Cameron claimed he was going to clamp down on "benefit tourism", for which the government conceded there was in fact no "quantitative evidence". He then announced migrants would no longer be able to claim out-of-work benefits for three months – which is effectively already the case. The depraved nadir of this migrant-baiting Dutch auction was reached when Ukip's leader, Nigel Farage, made clear that his call for Syrian refugees to be allowed into Britain would apply to Christians only.

In reality, the politicians are posturing because they can't control EU migration, but need a scapegoat for falling living standards, shrinking public services and the housing shortage. Faced with the electoral threat from Ukip, the Tories and their friends in the media have reached for the tried and tested alternative of blaming foreigners.

That will only strengthen Ukip's appeal. But it will also degrade social life and undermine economic recovery, as the government tries to restrict the right of British citizens to bring in a non-European spouse to those earning over £18,600 a year and clamps down on non-EU students. After years in which the explosive link between immigration and race has been partly defused, expect abuse of Roma people, Europe's most shamefully treated minority, to be ratcheted up.

That's far easier for the government and its supporters than dealing with the causes of the crisis through which people experience mass migration. As Damian Drăghici, Roma adviser to the Romanian prime minister, put it this week, Britain should be more worried about bankers "stealing billions" than "Roma begging in the street".

The growth of large-scale migration is after all part of the system of corporate globalisation that took hold in the past 30 years and widened inequality both within and between countries. It's also been fuelled by 15 years of western wars and intervention from Afghanistan to Somalia. And in eastern Europe, the exploitation and migration of low-waged and skilled workers has been central to the neoliberal model imposed after 1989.

It's that model that crashed in 2008 after years of stagnating real wages had fuelled the rise of the populist right across the continent. Public opposition to immigration in Britain isn't just a product of xenophobia or media mendacity, as sometimes claimed, but people's response to its impact on a deregulated labour market, under-invested housing and slashed public services.

In the past decade, European migration was used as a sort of 21st-century incomes policy in Britain as employers ruthlessly exploited migrant labour to hold down wages – which have since been cut in real terms for four years in a row as a result of the crisis.

The ready supply of low-cost migrant labour was only one factor in the earlier wage stagnation, which was driven by globalised trade, technology and the decline of unions. But the determination to fight anti-migrant bigotry and racism can lead some to romanticise deregulated migration as an undiluted good on whatever scale.

That's clearly not the case for either source countries, which can be stripped of skilled workers and professionals by richer states, or migrants subject to abuse and discrimination. Immigration rules for EU states, such as Britain, are incidentally heavily skewed in favour of white migrants.

For host countries, the overall economic impact of immigration may be positive, even if Britain's growth was relatively sluggish before the crash. And press and politicians' claims that migrants are a drain on the public finances is clearly nonsense. They are far less likely to claim benefits than those born in Britain and they make a large net contribution in taxes.

But the class impact is something else. Whatever the effect on average wages, there is clear evidence that lower-paid and unskilled workers' wages are often squeezed or cut by the exploitation of migrant workers in, say, construction or care work – while well-off professionals typically benefit from cheaper restaurants and domestic cleaners. And the competition for scarce housing and overstretched public services is greatest in the poorer areas where migrants tend to live.

That's why the policies that are desperately needed for the majority to break the grip of a failed economic model would also help make regulated migration work for all: stronger trade unions, a higher minimum wage, a shift from state-subsidised low pay to a living wage, a crash housing investment programme, a halt to cuts in public services, and an end to the outsourced race to the bottom in employment conditions. Those changes are necessary in themselves – but are also essential to draw the poison from immigration.

Twitter: @SeumasMilne