What is the biggest single drag on the beleaguered global economy? Opponents of globalisation might point to the current crisis, which shrank the world economy by about 5%. Proponents of globalisation might point to the remaining barriers to international flows of goods and capital, which also serve to shrink the world economy by approximately 5%. That sounds like a lot.

But the truly big fish are swimming elsewhere. The world impoverishes itself much more through blocking international migration than any other single class of international policy. A modest relaxation of barriers to human mobility between countries would bring more global economic prosperity than the total elimination of all remaining policy barriers to goods trade - every tariff, every quota - plus the elimination of every last restriction on the free movement of capital.

I document that remarkable fact in a new research paper. Large numbers of people wish to move permanently to another country – more than 40% of adults in the poorest quarter of nations. But most of them are either ineligible for any form of legal movement or face waiting lists of a decade or more. Those giant walls are a human creation, but cause more than just human harm: they hobble the global economy, costing the world roughly half its potential economic product.

The reason migration packs such economic punch is both simple and mysterious: a worker's economic productivity depends much more on location than skill. A taxi driver in Ethiopia's capital, no matter how talented and industrious, cannot earn more than a few thousand dollars a year. The same person doing the same job in New York City can easily earn $35,000 a year. The reason people will pay him that much is that his driving adds more than $35,000 of value to the New York economy, more value than his actions can add to the Ethiopian economy.

This has puzzled economists since Adam Smith in the 18th century. It is related to international differences in legal systems and geographic traits, and to pure proximity to other high-productivity workers. But regardless of the reason, the fact remains that simply changing a worker's location can massively enrich the world economy. And stopping such movement massively impoverishes it.

Stopping movement particularly impoverishes people born, through no choice of their own, in countries with little economic opportunity. The large majority of Haitians to emerge from destitution did so by leaving Haiti, not by anything they or any development agency did within the country. A low-skill male Cambodian can earn a living standard six times higher in the US than in Cambodia, for similar work. No act within Cambodia can reliably and quickly create so much opportunity for the industrious poor. And the benefits need not be limited to a tiny handful. In the late 19th century, roughly one third of Sweden's labour force permanently emigrated to opportunity; today, about half of Guyana has left Guyana.

How can the benefits of this - the world's greatest arbitrage opportunity - be reaped? There are numerous clear and sound proposals for more economically sensible migration policy. These include Lant Pritchett's proposals for bilateral guest-worker agreements, the ideas of Pia Orrenius and Madeline Zavodny for raising permanent economic visa allocations, and the proposal by Jesús Fernández-Huertas and Hillel Rapoport for tradable immigration quotas. I suggest using migration policy as one tool to assist people in poor countries struck by natural disaster. Each approach has advantages and disadvantages, but they have in common a drive to generate triple-wins for migrants, destination countries and origin countries, taking advantage of what Pritchett calls "the cliff at the border".

Many people fear that even a minor increase in international migration will wreck their own economies and societies. Those fears deserve a hearing. They are old fears, of the kind that filled US newspapers a century ago. The US population subsequently quadrupled, largely through immigration to already-settled areas. Today, even in crisis, America is the richest country in the world. History, too, deserves a hearing.

Of course, history is often the brooding and ignored stepchild of policy debate. Political constraints may make it impossible in the short-term to realise the gains from greater geographic worker mobility, just as political constraints blocked other forms of worker mobility in the past. But that can change. All the economic and social arguments against immigrant entry to the workforce could be - and were - deployed decades ago against female entry to the workforce. ("But men built those companies! Why should we allow women to work when there are qualified, unemployed men? Why should a man pay taxes for a woman's unemployment insurance? Will female employees assimilate and act just like men as we all wish? And what harm will be wrought in the homes they abandon?")

Now these arguments sound worse than ridiculous. Society decides who is or is not a member of the relevant club and, beyond the short-term, that decision can change massively. Though our fears will likely continue to impoverish us for some time, they need not do so forever.

• Michael A Clemens is a senior fellow at the Center for Global Development, where he leads the migration and development initiative