Thanks to an ever-growing number of international agreements, the taxes and tariffs that once slowed international trade are disappearing, making it easier to import, export, and do business across state lines. Increasingly, we do business in a borderless world.

But the very wealthy don’t just do business in a borderless world–they live in it. They have their own version of free trade agreements. They can and do buy their way into virtually any country they like.

There’s even a market for passports and residence permits that help wealthy types to move between countries. It’s not cheap to join the global elite, but it is relatively easy. That’s why the vote against the EU is so heartbreaking. It was the closest thing ordinary people had to the kind of border-hopping freedoms the ruling classes have long paid dearly for.

Herman Melville understood the distinction governing who versus what can move freely, long before “globalization” made its way into our vocabularies. “In this world, shipmates, sin that pays its way can travel freely, and without a passport,” preaches Father Mapple in Moby Dick’s famous whaling sermon, “whereas Virtue, if a pauper, is stopped at all frontiers.”

The EU, at least within its borders, is constructed to be different. Whether they’re sinful capitalists or virtuous paupers, Europeans can find a home, a job, even love in any of the union’s 28–for now–member states. When Britons voted to leave the EU, they justified their move by claiming they were voting against the city of London, against the “global elite,” and against the unfairly tiered rewards of globalization that favor those who are already wealthy.

“In this world, shipmates, sin that pays its way can travel freely, and without a passport,” preaches Father Mapple in Moby Dick’s famous whaling sermon, “whereas Virtue, if a pauper, is stopped at all frontiers.”

But in one crucial respect, their dissent was incoherent. These voters chose to leave in spite of the fact that the European Union is the only logically consistent arrangement of its kind–one that, in theory, lets goods and people move with equal freedom.

Hate won, and that’s a huge shame. The result risks reverting the UK back to the worst of both worlds: one where ordinary people can’t cross borders as they please, but their capital can.

To understand how unique the EU is when it comes to this sort of freedom, compare the single European market to other kinds of trade treaties. NAFTA, the North American free trade agreement struck in 1992, is great for industry. It removes trade barriers and tariffs on goods crossing two major borders. But it does not give all Mexicans and Canadians the automatic right to work in the US, or vice versa. (There are, however, short-term concessions for workers in very specific professions, and Canadian prime minister Justin Trudeau just announced that Mexicans could start visiting Canada without a visa.)

The Trans-Pacific Partnership will, once it is ratified, be similarly beneficial to businesses operating between, say, Peru and Brunei, or Japan and the United States. But it does not invite Chilean grocery store owners to fly to New York, or allow Japanese engineers to set up shop in Silicon Valley overnight.

These treaties aren’t as far-reaching as the EU. They aren’t political unions, and don’t have the EU’s open borders and common currency. They reveal a double standard. We’ll take your money, your goods, your huddled CEOs—but keep your people to yourself. The EU, to its credit, ironed out this central contradiction of globalization.

The Brexit vote was motivated by the twin anxieties of economics and identity. It’s easy to understand how those on the losing side of globalization game see EU policies as lowering their pay, threatening their jobs, and, most viscerally, turning their communities into fragmented factions they no longer recognize. It’s tempting to read this as a populist revolt against neoliberal economic policies, but it’s not a defeat of the status quo; not in theory, not in practice, not in the short or in the long term.

If this “Brexit” is truly based on worries about immigration and money, the actual Brexit strategy and the bilateral agreements that grow out of it are likely to keep promoting cross-border trade, but curbing the comings and goings of large groups of people. Banks based in London, for instance, currently employ a strategy known as “passporting,” which allows them to trade as though the EU was a single market. Will politicians in the UK and the EU, under pressure from powerful business interests, really end this practice–particularly in the face of lobbying from big banks? British politicians are already making promises not to, Boris Johnson and Jeremy Hunt among them.

This isn’t a populist revolution by any stretch of the imagination. It validates the hateful, xenophobic campaigning that culminated in the murder of MP Jo Cox. It does little to regulate big business or redistribute its spoils. Its effects on London and the “global elite” will be negligible. And it closes Britain and Britons off from the world.

The potential to live elsewhere, to travel, to immigrate and emigrate, is an invaluable part of the modern human experience. In my view, it is the single unequivocally positive thing about globalization. It is therefore our collective responsibility to extend this privilege to as many people as possible, and one that’s particularly relevant to a conversation that constantly invokes ideas about sovereignty and political freedom.

You don’t have to be a hardcore libertarian or an internationalist Marxist to believe that it’s a good thing for people to live where they like. You just have to have moved once in your life, or to have wanted to. Now, the great irony is that those who voted “Remain” may want to leave very badly–but have no place left to go.