New York is a cosmopolitan place. London is of another order.

Consider the financial services sector, which accounts for more than 11% of the UK government’s tax receipts: the Bank of England is run by a Canadian, a Frenchman is head of the London Stock Exchange, executives from Portugal and New Zealand run two of the country’s state-run banks, and Barclays, one of the few big British banks left with global aspirations, recently replaced a Brit with an American at the helm.

In both New York and London, more than a third of the population is foreign born. But in London’s financial district, foreigners run the show. They even have a term for it: “Wimbledonization”, derived from the fact that England hosts the best players from around the world at its great tennis championship, but a local rarely wins it.

“We’re not bothered by who owns it or who runs it,” just that it takes place here, Mark Boleat, policy chairman for the City of London Corporation, told me a few years ago, explaining the term. “If a foreigner can run it better than we can, that’s great,” he said. “We’re different from other countries in this way.”

This reflects either a remarkable spirit of generosity or a lack of local talent. If the latter, the country’s decision to vote itself out of the European Union, in large part to limit the free movement of people from the bloc, is bad news for for London’s hitherto dominant finance industry.

“It’s another complication on their list of hell,” says John Burr, a partner at Boyden, an executive search firm.

Before Brexit, banks were already suffering from stricter regulations, stiffer capital requirements, and low interest rates, all of which dent profits. Brexit is another blow—shares of Barclays, Lloyds, and RBS have all fallen by more than 20% since the vote. Officials elsewhere in the EU are eyeing up the activities their financial capitals can take from London (paywall).

Without preferential access to EU markets, banks have hinted that they could move thousands of bankers out of London. Finance firms’ biggest costs are their people, and they are not sentimental about adjusting this “cost base” quickly to suit prevailing conditions.

But if London becomes less attractive, it is unclear where banks would prefer to put people: Germany has some of the toughest labor laws in the world, and France taxes people at “confiscatory” levels (the words of one financial services executive). It’s also worth noting that banks have recently been laying people off more than hiring: only regulatory and compliance divisions are growing.

One senior Wall Street executive, whose bank has hundreds of people in London, said, “We’re not going to hire, and if business slows, we will start shrinking.” If Britain goes ahead with severing ties with the EU, he predicted that his London office—now the company’s international HQ—would become a much less important satellite office, like outposts in Germany or Italy. If others make similar decisions, the departures have a multiplier effect on forgone tax revenues and lower spending.

Ironically, Brexit might inadvertently achieve the aims of its staunchest supporters, though perhaps not in the way they intended. Immigration to Britain has surged because the economy was among the strongest in Europe, with low unemployment and dynamic, world-beating industry clusters. With the odds of recession rising, and markets in turmoil, it’s fair to say some of those advantages will wane, and with them, immigration.