The comedian and TV presenter Griff Rhys Jones is ready to quit the UK in protest at plans by the Labour party to introduce a mansion tax if it wins the 2015 general election. As the Telegraph reports:

“He himself lives in a ‘gigantic’ house in a part of central London that was, when he bought it 15 years ago, a ‘slum’. He has a track record of buying large, run-down properties and turning them into homes for himself and his wife, Jo. His Fitzrovia house has appreciated so significantly that he is contemplating moving overseas if Labour wins the election and introduces a mansion tax.”

“It would mean I’d be paying the most colossal tax, which is obviously aimed at foreigners who have apparently come in and bought up all the property in London,” he says. “That sounds about as fatuous an idea as that immigrants are stealing all the jobs. I’d probably go and live abroad because I could get some massive palace which I could restore there.”

There has been a lot of nonsense talked about the mansion tax. This, from the chief executive of Legal & General, is typical.

“People who choose to prioritise buying a home have typically made sacrifices to do so: fewer foreign holidays, meals out or other luxuries. Through no fault of their own, their prudence would be punished by a mansion tax.”

The idea that folk who own houses worth millions have made sacrifices, saved hard or been prudent may well be true (at least for some). But that sacrifice, saving and prudence is not what has been responsible for their homes being worth so much money. The inflated price of houses in many parts of the UK is a consequence of scarcity and a lax fiscal regime. In most cases, the financial gains made by homeowners are only in very small part due to their own efforts (for example, insulating or other improvements). The vast majority of the gains are as a consequence of rising land values.

Labour has yet to spell out the details of its plans but they involve a levy on properties worth over £2m. Ed Balls announced the policy in an article in the Evening Standard last month. The Financial Times calculated that on average, the owners of properties worth over £3m would pay an average of £19,000 per year.

Facebook Twitter Pinterest Griff Rhys Jones lives in a house apparently worth at least £7m in Fitzroy Square. Photograph: CandyAppleRed Images/Alamy

Griff Rhys Jones and his wife own a property in Fitzroy Square in the London borough of Camden. They bought the property for £1.45m in 1998 after Camden council granted planning consent for a change of use from offices to a residential home (see Land Registry title). The couple then did renovations and now have a home with seven bedrooms, three bathrooms and four reception rooms.

According to Zoopla, the property is worth £7,012,156 and has risen in value by £3,015,251 over the past five years. The rental value is estimated at £16,167 per month (£194,004 per year). Rhys Jones currently pays £2,640.96 in council tax to Camden council.

Assuming that £1m was spent undertaking renovations, the Rhys Joneses have seen their property rise in value by around £4.5m. That sum is unearned increment (economic rent in economic theory) and, since principal residential properties are exempt from capital gains tax, the gain is entirely tax-free. This tax relief is worth an estimated £10.4bn per year to homeowners according to the National Audit Office. Successive governments have put in place a fiscal regime for domestic property that allows Rhys Jones to make a £4.5m tax-free capital gain without any effort on his part.

A sensible system of recurrent taxation would be designed to curtail such asset inflation by socialising this rent rather than allowing it to be appropriated tax-free by private interests. The mansion tax is a badly designed tax. As the Institute of Fiscal Studies commented in February 2013: “Rather than adding a mansion tax on top of an unreformed and deficient council tax, it would be better to reform council tax itself to make it proportional to current property values.”

If property taxation were properly proportional and the Rhys Joneses paid the percentage rate (1.85%) that a mid-point English Band D property is liable for, then they would be paying £129,724 per year. The mansion tax is liable to be about a tenth of that.

That kind of liability would deter most buyers who, as a consequence would offer less for the property so as to pay less in annual holding costs – which is precisely what a well-designed system of recurrent property taxation would do. Lower property prices would mean less indebtedness and more resources invested in the productive economy. But that is not the kind of economy that either Labour or the Tories appear to be interested in.

In the meantime perhaps Rhys Jones should be grateful.

• This article originally appeared on Andy Wightman’s blog