On Thursday, voters in Britain will go to the polls to decide whether to stay in the European Union or exit — the "Brexit" option.

For many Brits, the debate isn’t just about Britain’s economic future — it’s a culture war over British identity. Many supporters see leaving the EU as essential to safeguarding Britain’s distinctive culture and political institutions, while opponents have portrayed the exit campaign as nativist and intolerant. This debate became even more heated after a member of Parliament was murdered on Thursday by a man with alleged ties to white supremacist groups.

And while many Brexit supporters say they’re concerned about immigration, this is actually a shorthand for a larger set of social anxieties held by many Britons — especially older voters in outlying parts of the UK. The post-industrial economy has delivered an economic boom for some British people and stagnation for others. Many Britons feel anxious about these broader shifts, and immigrants make convenient scapegoats.

Many complaints about immigration aren’t really about immigration

One of the things that makes the Brexit debate so divisive is that although the British population as a whole is split close to evenly, this doesn’t reflect an even split in every subgroup. In fact, there’s a massive amount of demographic "shape" to this referendum – the "remain" camp has large leads in people under 35, and among the richer and university-educated, while "leave" tends to win unskilled and semi-skilled workers. Older voters back "leave" by a wide margin.

There’s also a geographical pattern: Coastal areas are very Euroskeptic, partly due to a large retired population, while cities are in favor of staying in.

A key part of the appeal to the older demographic, if poll results are to be believed, is complaints about National Health Service doctors’ surgeries being clogged up with immigrants. The Leave campaign certainly seems to think immigration is its strongest issue, and the polling seems to agree.

Except that in this case, "concerns over immigration" aren’t necessarily about immigration. The regions that are strongest supporters of Brexit, the anti-immigration UK Independence Party, and anti-immigration politics in general tend to be areas that don’t actually have all that many immigrants. It is doubtful that immigration (which tends to be of young and healthy working people) can be putting much strain on the health or education systems.

It’s equally doubtful that immigrants are pricing locals out of the housing market in sleepy British towns like Stoke-on-Trent. So why is there so much resentment of the small number who are there?

Economic change is dividing Britain into winners and losers

One big factor is that the character of a neighborhood is changed by the kind of people who move out just as much as by the kind of people who move in. And regional migration within the UK has been a big phenomenon over the past 20 years, as London and the Southeast have grown – economically and in population terms – much faster than the rest of the country.

Some of the most Euroskeptical areas have seen rapid declines in their population. In Stoke-on-Trent, for example, the local council has so many empty homes that it sometimes sells them for £1 each, in a publicity stunt aimed at attracting economically active people.

This depopulation has been driven by deindustrialization. While the UK has in general been good at creating jobs and keeping unemployment down, the northern and coastal towns where Brexit support is the highest tend to be home to either difficult and dangerous labor-intensive industries or low-level service industry employers like hotels and care homes. As the UK labor force has got more productive and better educated, on average, it’s unsurprising that workers have tended to migrate to the higher-paid new jobs being created in the growing regions around London.

Lopsided and London-centric development is a real problem. And of course, although it is an old cliché for miners and fishermen to hope that their children don’t follow them, that doesn’t make it any less painful to have your children (and now grandchildren) living on the other side of the country, in a world that’s culturally and economically even further away than its geographic distance.

It’s particularly challenging when the housing and employment vacuum created by their departure gets filled with new European immigrants, who act as very visible symbols of the underlying change when you interact with them in their service jobs, or at the doctor’s office.

Restricting immigration won’t do what advocates think it will

Restricting immigration wouldn’t actually solve many of the problems that are being attributed to immigration. The health and education services have a budget set according to estimated population and its growth; if the immigrants are shut out, the budget will be cut accordingly and appointments will be just as difficult to get.

That’s actually the best-case scenario. Many immigrants are net taxpayers. Removing them might worsen the UK’s fiscal position and force even deeper cuts to public services.

Without immigration, it’s likely that the last remnants of industry in the coastal decline belt would close down

And it’s equally unlikely that clearing the Lithuanians and Portuguese out of the chicken processing factories would mean that British workers would start pouring back out of London, eager to go back to their old home towns to slit gizzards and hose down abattoir floors. With 5 percent unemployment, the UK does not have spare workers; that’s why such a high proportion of newly created jobs go to overseas citizens. Without immigration, it’s more likely that the last remnants of industry in the coastal decline belt would just close down.

It’s understandable that people react to what’s in front of their faces. But as well as being a somewhat mean way to deal with the world, reacting to economic decline by limiting immigration is horribly counterproductive. The problem for places like Grimsby or Merthyr Tydfil isn’t that they’re full of immigrants – it’s that so many of the young people have left.

Dan Davies is a senior research adviser for Frontline Analysts, a global research outsourcing firm. He writes in a personal capacity.