There is a building in the Cayman Islands that is home to 12,000 corporations. It must be a very big building. Or a very big tax scam. Tax havens are in the spotlight since the Chancellor, George Osborne, did a deal the other day with the Swiss authorities to slap a levy on secret bank accounts held there by British citizens. Opinions are divided on the move, which could net the Treasury £5bn, but which tacitly legitimises bank accounts kept secret from the Inland Revenue. It is a de facto amnesty for those guilty of tax evasion crimes. And they will pay less than they would if they declared their income to the British taxman.

Are there any legitimate reasons why anyone would want to have a secret bank account – and pay a premium to maintain their anonymity – or move their money to one of the pink dots on the map which are the final remnants of the British empire: the Caymans, Bermuda, the Turks and Caicos and the British Virgin Islands?

The moral case against is clear enough. Tax havens epitomise unfairness, cheating and injustice. They replace the old morality embodied in the Golden Rule of reciprocity – that we should do as we would be done by – with a new version that insists that those who have the gold make the rules.

The old view, the neocon American Christopher Caldwell wrote recently, subscribes to a religious understanding of money that was universal in the Christian world before the rise of Protestantism, which acknowledges that people are alive but money is not, making it wrong for the latter to take precedence over the former – a notion as outdated as usury, he suggested tartly.

But what is the moral case for tax havens? We can dispense with the argument advanced by their administrators that if they didn't take the money it would simply move to more distant locations; that is the self-serving logic of a man who sells torture equipment to an oppressive regime. Apologists insist that tax havens protect individual liberty. They promote the accumulation of capital, fair competition between nations and better tax law elsewhere in the world. They also foster economic growth. So much so, the Institute of Directors has said, that Britain should not curb tax havens but emulate them, promoting the growth of more hedge funds in the UK.

Yet even if all that were true – and it is not – does it outweigh the ethical harm they do? The numbered bank accounts of tax havens are notoriously sanctuaries for the spoils of theft, fraud, bribery, terrorism, drug-dealing, illegal betting, money-laundering and plunder by Arab despots such as Gaddafi, Mubarak and Ben Ali, all of whom had Swiss accounts frozen.

The corruption spreads contagion, as the financial writer Nicholas Shaxson showed in Treasure Islands, his book about offshore finance which exposed secrecy, corruption and intimidation in places as seemingly innocent as that land of milk and money, such as Jersey in the Channel Islands.

But the moral bankruptcy of the tax haven runs deeper. Indeed it is intrinsic to its purpose. The British Virgin Islands is the global capital for the incorporation of offshore companies. Though it has a population of just 22,000, it has 823,502 registered companies which make vast amounts of money through the wonder of transfer pricing. It works like this. Suppose I manufacture a product in Africa and sell it in the UK. If I am a canny businessman I set up an intermediate company in a tax haven. It need do nothing except exist on paper. But through it I can buy all the products I make in Africa, dirt cheap, and then sell them, at a much higher cost, to my UK subsidiary. The African and British companies do not, thus, make much profit, so I have little or no tax to pay. All the money stays offshore, where taxes are low or non-existent. This is perfectly legal. But it distorts the world economy and means I pay no tax. I can also borrow where rates are lowest and keep my costs where they are most tax deductible.

That is why General Electric paid no taxes in 2010, despite $14.2bn profits. It's why Barclays, with 181 subsidiaries registered in the Caymans, paid relatively little UK tax on its worldwide profits. Rupert Murdoch's News Corp, with 152 subsidiaries in tax havens according to the US government, paid no net UK corporation tax between 1988 and 1999.

Half the world's trade flows through tax havens. Every multinational uses them routinely. So do banks. Almost 70 per cent of international trade now happens within, rather than between, multinationals. Christian Aid reckons that tax dodging costs developing countries at least $160bn a year – far more than they receive in aid. The US research centre Integrity estimated that more than $1.2trn drained out of poor countries illicitly in 2008 alone.

Tax injustice is systemic to the tax haven. Barack Obama once understood that. During his election campaign he promised to crack down on corporate loopholes and tax havens. But he and other world leaders have not delivered on bursting open the seedy secret underworld of tax havens that nurtured the hedge funds, derivative trading and off-balance sheet lending that fuelled the 2008 global financial crash.

Their malign influence continues, with hedge funds accounting for at least 30 per cent, and perhaps as much as 60 per cent, of current trading on the London and New York exchanges. There, they have quintupled short-selling. They have turned credit default swaps, designed as a protective insurance, into a way of betting on the failure of a company. The Caymans (population 50,000) is home to 70 per cent of hedge-fund registrations worldwide.

And, as rich people waive their taxes, poor people wave goodbye to their jobs. "The rich are different from you and me," Scott Fitzgerald famously said. "Yes," wisecracked Ernest Hemingway in response, "they have more money." Today the difference is that they pay less in taxes.

The real shame of Osborne's half-baked deal with Switzerland is that it has undermined the revised EU savings tax directive. That would have required an automatic exchange of information on income in bank accounts throughout the EU and in Switzerland, Lichtenstein and Britain's tax havens. All the EU member states, but two, had approved it. It would have dealt not just with individuals but also with companies, trusts, foundations and other complex structures.

Some say an attack on tax havens is an attack on wealth creation. It is no such thing. It is a demand for the good functioning of capitalism, balancing the demands of efficiency and of justice, and placing a value on social harmony.