Article 50 and Brexit loom. “Taking back control” starts with immigration control, ending free movement inside the European Union. For the Remain resistance movement, that freedom has been a fundamental principle, to be defended to the last.

All my instincts are to defend the freedom to work, study and retire anywhere within the EU. I was (and am) a Remainer. I value diversity and have a diverse multi-ethnic family. I have spent half a century campaigning against anti-immigrant prejudice, from Enoch Powell’s “rivers of blood” to the Turkish “hordes” of the referendum campaign.

I spent five years as secretary of state battling the Tories’ foolish net immigration target and damaging restrictions on overseas students and workers. As a liberal economist, I welcome freer trade and globalisation in general; and as a political liberal I oppose attempts to fence people in. I naturally value the freedom to travel around Europe for business or pleasure with minimal restriction.

But I have serious doubts that EU free movement is tenable or even desirable. First, the freedom is not a universal right, but selective. It does not apply to Indians, Jamaicans, Americans or Australians. They face complex and often harsh visa restrictions. One uncomfortable feature of the referendum was the large Brexit vote among British Asians, many of whom resented the contrast between the restrictions they face and the welcome mat laid out for Poles and Romanians.

British opposition to immigration is mainly colour-blind. Until well into the 1990s “immigration” was code word for race. But in recent years concerns over immigration have been precisely that. The question has become: is unrestricted immigration – albeit from some countries only – desirable?

The economics are ambiguous. Seen globally, more migration is undeniably a positive. People moving from high unemployment, low productivity countries to areas of labour scarcity and higher productivity produce economic gains. But the benefits accrue mainly to migrants themselves (and business owners). For the receiving country, the benefits are less obvious: a bigger economy but not necessarily a richer one. Immigrants may be more productive than indigenous workers but they have dependents too. They are usually young people and therefore likely to be more flexible, more mobile and more likely to work contributing more in tax than they take out in benefits and subsidised services. But they grow older so these benefits are non-recurring.

There are also distributional effects. Critics complain that immigrant workers depress wages and reduce job opportunities for natives. Undoubtedly, this happens in some occupations, like building and taxi driving. But there are other areas where immigrants are not competing and bring complementary skills, creating jobs. When I was secretary of state I commissioned studies to evaluate this. The conclusions were sufficiently reassuring that the Home Office blocked my department’s wish to publish them. Losers, however, there undoubtedly are.

The economic arguments are not conclusive but, on balance, favour some net migration of younger, skilled workers. More liberal Brexiteers concede that point. It is also reasonable for Remainers to accept that there should be controls, as for non-EU migrants. That is also where public opinion is. Long-term social survey analysis suggests that the demand for effective immigration control coexists with greater tolerance of diversity.

There is no great argument of liberal principle for free EU movement; the economics is debatable; and the politics is conclusively hostile. The argument for free movement has become tactical: it is part of a package that also contains the wider economic benefits of the single market. Those benefits are real, which is why the government must prioritise single market access and shared regulation. Yet that may not be possible to reconcile with restrictions on movement. The second-best option is customs union status, essential for supply chain industries.

I do not see much upside in Brexit, but one is the opportunity for a more rational immigration policy. First, it will involve legitimising the position of EU nationals already here. It must involve a more sensible way of dealing with overseas students, who are not immigrants and benefit the UK. The permeability of the Irish border must lead to a united Ireland in Europe. And, not least, there can be a narrative in which control on labour movements is matched by control on capital – halting the takeovers that suffocate the innovative companies on which the country’s future depends.