The Brexit vote will force the next U.K. prime minister to make a significant choice: Whether to control immigration into the country, or preserve the pre-eminence of the country’s financial center, known as the City of London. Doing both appears to be infeasible.

That’s because membership in the European Union, which British voters rejected, is founded on four freedoms guaranteed to EU citizens and firms: free movement of goods, of capital and of people, along with the right to offer services and establish businesses anywhere in the union. As the EU leaders reiterated on Wednesday, after their first meeting that excluded the U.K. from their conversations, access to the EU market “requires acceptance of all four EU freedoms — including freedom of movement. There can be no single market à la carte.”

Leaders of the Brexit movement have said they want access to the market without allowing free movement of people onto British shores. But that cafeteria approach, allowing the U.K. to pick and choose the ingredients of a new relationship, is off the table. This can hardly be surprising; if the EU were to cave in to this demand, many other countries would immediately insist on getting their own customized deals. The great achievement of the single European market would quickly unravel.

What does the referendum have to do with London being the top-rated financial services location in the world? Right now about 1.4 percent of the U.K. workforce is employed in the City. Most of these jobs are high-paying, and the financial services industry accounts for about 11 percent of all tax revenue in the economy. So regardless of one’s views about bankers, the contribution to the public purse is substantial, and if the City were to shrink the consequences for everyone in the U.K. would be serious.

What is less well known is that the City’s strength depends in part on the U.K.’s membership in the EU. Because the U.K. has been an EU member, U.K.-based companies have the right to run a pan-EU business in London without interference. This right, known as “passporting,” guarantees a level playing field for U.K. firms, provided that their businesses are in compliance with the relevant EU regulations. Without passporting privileges, a business seeking to operate across multiple EU states would have to contact the relevant host country regulators to secure approval to proceed. The current passporting rights are therefore essential to the agglomeration benefits that make London such an attractive location. Right now London-based firms get all of the amenities of being in London without running into country-specific roadblocks from other EU members.

RANK CITY POINTS CHANGE IN RANK 1 London 800 — 2 New York 792 — 3 Singapore 755 ▲ 1 4 Hong Kong 753 ▼ 1 5 Tokyo 728 — 6 Zurich 714 ▲ 1 7 Washington, D.C. 712 ▲ 3 8 San Francisco 711 ▲ 1 9 Boston 709 ▲ 3 10 Toronto 707 ▼ 2 11 Chicago 706 — 12 Seoul 705 ▼ 6 13 Dubai 699 ▲ 3 14 Luxembourg 698 ▲ 5 15 Geneva 694 ▼ 2 16 Shanghai 693 ▲ 5 17 Sydney 692 ▼ 2 18 Frankfurt 689 ▼ 4 19 Shenzhen 688 ▲ 4 20 Osaka 687 — 21 Montreal 686 ▼ 4 22 Vancouver 684 ▼ 4 23 Beijing 682 ▲ 6 24 Taipei 677 ▲ 2 25 Tel Aviv 676 — 2016’s most important global financial centers and their change in rank from 2015 Points, on a 0 to 1,000 scale, are based on financial indexes and industry assessments Source: Global Financial Centers Index

Starting in the 1990s, when the U.K. was deciding whether to abandon the pound in favor of the euro, it has been the policy of successive U.K. governments to make sure that the relationship with Europe supports a robust role for the U.K. financial services industry. Indeed, before the referendum, the top U.K. official in the EU was Jonathan Hill, the commissioner for financial services. It is not an accident that he had this position. The U.K. has fought continuously to maintain EU rules that were acceptable for London. One noteworthy example was in 2011, when the U.K. vetoed an EU budget deal that was acceptable to all 27 other members.

In order for the U.K. to maintain the status quo regarding the City, it therefore will need to secure passporting rights. However, it would seem the only way that is achievable is if the U.K. accepts the four freedoms, which means no restrictions on the movement of people.

Once the negotiations over the new relationship between the U.K. and EU commence, the most efficient arrangement for the U.K. would be to seek a deal similar to those that the EU has with Norway and Switzerland, the two other big European countries outside of the union. These deals give the non-EU countries access to the single market in exchange for meeting certain conditions. The deal with Norway and the EU is simple. Norway agrees to comply with the four freedoms but is allowed some latitude to deviate from some secondary EU priorities. Norwegian financial services firms enjoy passporting rights, but Norway pays into the EU budget and has no seat at the table in forming EU policies. The current EU arrangement that the U.K. voters just rejected is arguably better than the one Norway has.

An alternative would be the one that Switzerland and the EU have struck. This arrangement, implemented through a series of many bilateral treaties, again requires a contribution by Switzerland to the EU budget and includes the free movement of people, though there is some wiggle room on this front. For instance, an EU citizen seeking residency in Switzerland must have a valid employment contract. Swiss financial services do not have full passporting rights. So it is possible that this could be a model on which the U.K. could seek to build that might allow its leaders to reduce immigration. Going down this road sacrifices passporting privileges.

There are some benefits of immigration that drew little attention in debate that preceded the referendum. In the coming years, the U.K. will face high pension costs as current workers retire, similar to the surge in baby-boomer retirements in the U.S. The easiest way to mitigate those costs would be through the wages and taxes of new immigrant workers. The U.K. might choose to highlight this issue in explaining a decision to sign on to a deal with relatively few limitations on immigration.

But the chances the U.K. will try to ignore the referendum and retain the status quo seem slim. Theresa May, the odds-on favorite to become the next Conservative Party leader (and hence prime minister), has already said “Brexit means Brexit” and has ruled out this strategy. Given that the Conservative Party had committed to this idea repeatedly in advance of the election, this should not be surprising.

And there are many in the EU who would like to take advantage of any decline in London’s status. EU officials would be happy to claw back as many of the high-paying financial services jobs and tax revenues from the U.K. as possible. The stakes here are large. To take just one example, swap contracts that are denominated in euros are a trillion-dollar market, and most of that business is now centered in London. French President Francois Hollande has said that this activity should be moved back into the EU.

The U.K. might conclude that the combination of its strong legal system, which makes dispute resolution relatively easy, along with its flexible labor market, which makes it possible to quickly downsize a business, will be sufficient to keep the City intact. However, this is far from assured. A decision to clamp down on the free movement of people could be the first step toward the financial fragmentation of Europe.

CORRECTION (July 5, 2 p.m.): An earlier version of this article misidentified one of the countries outside the EU that have financial arrangements with the union. It was Switzerland, not Sweden. (Sweden is a member of the EU.)