For some people it is all absolutely clear. By indicating that he will somehow try to limit migration to the UK from the rest of the European Union, David Cameron has moved closer than ever to embracing what we now call Brexit – the British exit from the EU. In doing so, he has shown that he is in thrall to mad-eyed Tory Europhobes, who in turn are doing the work of those dastardly merchants of racist calypso, Ukip. In this view, the free movement of people is not just a fundamental part of our membership of the EU, but modernity writ large: anyone who questions it must be off their rocker.

In fact, the issue is a bit more complicated than that. Some elements of Tory posturing on Europe – their manoeuvrings on human rights, for example – say much more about them than the wider public. But free movement has not arrived in the foreground of politics solely because of Conservative loopiness. It is there thanks to the way that even a dysfunctional democracy such as ours works: because it is a huge issue, and millions don’t like it. For the sake of Britain remaining in the EU, they will probably have to lump it, and this option will be by far the best for the country as a whole.

There is plenty of sense in the idea that an ageing UK population will increasingly need the help of migrant labour, that whole swathes of the economy would shrivel without their contribution, and that the tensions surrounding EU migration may well subside over time. The point is: before anyone joins this increasingly testy and often ugly debate, the least they should do is admit that this is very difficult stuff to deal with.

If the troubled modern politics of the EU is reducible to any single factor, it is surely wild asymmetry. The arrival of the eurozone crisis demonstrated the folly of subjecting very different national economies to the same monetary regime. And though migration within Europe had long been an accepted part of life in the EU, the noble ideal of free movement eventually collided with the same economic imbalances – in 2004, when the accession of eight former eastern bloc (or A8) countries put a rocket under free movement, and Britain decided against transitional controls. In retrospect, that was one of those choices that reflected the Blair/Brown period’s awful mixture of complacency and detachment from reality.

The debate about migration is too often reduced to cold numbers, but they have their uses. In 2003, the average wage in Latvia, the poorest accession country, was just one eighth of that of the 15 existing member countries of the EU. The ensuing years have seen a modest increase, but the gaps between former eastern bloc countries and the established EU nations are still huge. The monthly minimum wage in Poland, for example, comes in at just over €405; in Latvia it is €320, while in Lithuania it is €300. The figure for the UK is €1,420 – which, for those who qualify, will be boosted by such benefits as tax credits. In that scenario, movement will tend to be in only one direction – which may gladden the hearts of those who rejoice in diversity but will also, inevitably, sow resentment and tension.

The University of Oxford’s Migration Laboratory estimates that between 1991 and 2003 about 61,000 migrants from the wider EU came to the UK each year. Between 2004 and 2012, by contrast, that number almost tripled, to 170,000 annually. The 2011 census put the number of UK residents from Poland alone at 654,000.

On the outer edge of the migration experience are such places as Boston in Lincolnshire and Wisbech in Cambridgeshire: towns whose very fabric has been put at the mercy of capitalism at its most Darwinian and an economy that, via gangmasters and rogue landlords, blurs into criminality (and in which, contrary to rightwing stereotype, 99.9% of migrants are the victims, not the perpetrators).

This year I visited Wisbech – where a third of the 30,000 population is now estimated to be from overseas – and what was happening there spoke loud truths about why free movement has become so politicised. For all that recently arrived families have started to settle, and their children are acquiring new, hybrid identities, there are still glaring problems. Young men from eastern Europe often live four or five to a room, and work impossibly long hours; with echoes of Europe’s macroeconomic asymmetries, the local labour market is divided between insufficient jobs that be can be done by people with families and mortgages, and a surfeit of opportunities for those who will work whenever they are required for a relative pittance.

This creates endless tension. There have also been inevitable problems surrounding how far schools and doctors’ surgeries have been stretched. Is anyone surprised? Moreover, even if such places represent socioeconomic extremes, similar problems surface whenever large-scale migration fuses with the more precarious parts of the economy. In modern Britain, this obviously happens often, and the under-reported consequences of austerity have hardly helped.

What passes for the modern left tends to be far too blase about all this. Perhaps those who reduce people’s worries and fears to mere bigotry should go back to first principles, and consider whether, in such laissez-faire conditions, free movement has been of most benefit to capital or labour. They might also think about the dread spectacle of people from upscale London postcodes passing judgment on people who experience large-scale migration as something real.

The numbers go up and down, but current polling suggests that nearly 46% of people in the UK oppose the principle of free movement. The issue surely sits at the heart of why as many as 47% of us would give our assent to Britain committing economic hara-kiri and leaving the EU altogether. Taken together, those figures testify to two things: how close Britain now stands to the EU exit door, and how awkwardly the dream of the single market sits with fundamental features of the UK economy that tend to leave people scared out of their wits. Until that aspect of our national condition changes, our angst about Europe will continue.

For the moment the prime minister frets about Nigel Farage, but also feels the searing heat of public opinion. Labour’s Ed Balls, too, has recently been heard floating the idea that some kind of treaty change may soon be necessary, so that people understand that free movement “is not a free-for-all, and that it is being managed”. Particularly when it comes to the Tories, such twisting and turning looks not just cynical but devoid of any meaningful answers to a whole host of problems.

There again, do the shrill voices accusing them of pandering to prejudice have any convincing stance of their own? Or is the fashionable metropolitan option still to cast aspersions on millions of people, and then look the other way?