I sympathise with Jimmy Carr. He makes it from Slough to Cambridge, cracks a few jokes and finds that two plus two equals ten after tax. Then along come some posh geezers crying, "Oh I say, that Carr chap may be legal but he's morally repugnant. Let's duck him in the bog."

My accountant once informed me that I too could inherit the earth, or at least deny a fragment of it to the taxman. He said I should become a company and pay tax only on what I took out of it. I soon discovered that I was taking everything out of it, and any saving was cancelled by fees. I also felt something unknown to Carr, Ken Livingstone, government officials and BBC executives. These were dark arts and I was doing something dodgy.

This week the wordsmiths have been dragged from their attics to define "dodgy". How does it fare against nifty, fishy, tight-fisted, fleet-footed, totally baffling or morally repugnant? What is evasion as against avoidance, ineffective against illegal, must against ought? Meanwhile the money followed the travel supplements, to such exotic climes as Bermuda, Gibraltar, Grand Cayman, Monaco, Luxembourg and Liechtenstein. There are apparently more comedians in Jersey than in Blackpool, more film-makers in the Virgins than in Hollywood.

From the moment of taking office, coalition ministers have been scathing about bank bonuses, top salaries and tax dodgers. George Osborne says he regards "tax evasion and indeed aggressive tax avoidance as morally repugnant". Cameron criticises Carr for stuffing money "into something where he doesn't pay tax". The industry secretary, Vince Cable, froths at the mouth at "neither sustainable nor justifiable" bank bonuses.

Yet these fine words are nothing when confronting the full fury of big money. We are left with a moral quagmire. The rich are "entitled" to arrange their money according to the law, and we can merely condemn them as immoral. Wealthy executives claim the money they have filched from their shareholders and stashed offshore is "small in the totality of things". Yet revelations in the Times this week suggest that over £5bn a year may be lost to the exchequer, or over 1p on the standard rate of income tax. This should end British smugness at the fiscal shortcomings of Greeks and Italians.

Benjamin Franklin declared that the only certainties in life were "death and taxes". He reckoned without the London accountancy profession. An adviser promoting the avoidance scheme used by Take That told a Times undercover reporter that the band "don't like paying the taxman". Nor does anyone. Some Conservatives celebrate "tax freedom day", the moment each year when they say we stop working for the government and start working for ourselves. It has crept onward to June. They perhaps agree with the American philosopher Robert Nozick that taxation is state theft, justified only for national security.

Modern tax theory is rooted in more than revenue raising. It adheres to fairness, redistribution and incentive. For most people, paying taxes defines their citizenship. Whether or not avoiders are in the same class as "benefit cheats" is moot: most cheats have legal "sickies" as most film companies have brass plates in the Caymans. What is surely beyond dispute is that refusing to pay taxes in a shared community is anti-social, unfair and infuriating.

To add irony to injury, those who seem most obsessed with avoiding tax are, after financiers, people in the arts and sport – some of whom, like Carr, proclaim liberal sentiments. The biggest scams reported this week were introduced for the film industry by Tony Blair and Gordon Brown. They were to "drive cool Britannia" with glitzy parties in Downing Street and Cannes. This cool is estimated to have saddled the exchequer with an accumulated £5bn in tax costs.

Two factors underpin the scams. The first is the reluctance of British ministers of both parties to upset the metropolitan rich. They wine and dine with non-doms and off-shores. For a decade they have drawn their aides from those about whose wealth Lord Mandelson was "intensely relaxed". This same financial establishment now lobbies for cash support for banks – but not for high streets. It is the sea in which all parties swim.

Less explicable has been the reluctance to grasp the single institution most useful to tax avoidance, the off-shore haven. The British Virgin Islands now have 30,000 citizens and 457,000 companies. The convicted arms smuggler Viktor Bout used a front based in Gibraltar, reportedly a centre of drug money laundering. America claims that 75% of the world's hedge funds are "located in" the British Caymans, with deposits bigger than the whole of New York. The music download capital of the world is Luxembourg, which boasts the headquarters of Amazon, Apple and Vodafone.

The island of Jersey claims British citizenship and loyalty while inviting thousands of tax-dodgers to deny the crown the revenues due to it. These places protest they are accidental beneficiaries of globalisation, yet they advertise themselves as tax havens and live off the wealth. The US Senate calculated in 2008 that as much as $7tn of American bank deposits were located in British "crown colonies", dodging some $40bn in revenues.

There is no moral or legal justification for individuals or companies doing business in Britain to avoid paying tax by buying a nameplate overseas. The claim to offer tangential benefits to Britain is like a non-dom tycoon protesting that he supports his local church and cancer charity.

In 2009 the G20 declared its intention to "close the door on tax havens", but did nothing about it beyond requesting more transparency. The role of a tiny state such as Luxembourg is particularly outrageous, claiming to be "part of Europe" when it suits it, yet "offshore" when not. Modern governments appear to agree with the New York heiress Leona Helmsley: "It is the little people who pay taxes."

There is nothing most of us can do about these scams except look on in amazement and howl. Like a gang of terrifying hoodies charging down the high street, tax dodgers protest they are "doing nothing illegal". I am sure that is usually true. But no such let-out is available to politicians, who are the arbiters of legality. They may excoriate tax avoidance, but they are the only people in a position to stop it.

• Follow Comment is free on Twitter @commentisfree