There are about 65 million people in the UK and 60 million acres of land – almost enough, in theory, for an acre each. (It’s not quite that simple, of course: not all acres are equal.) Yet about two-thirds of the land – 40 million acres – is owned by fewer than 6,000 people. If there is a more telling statistic about the unequal distribution of wealth in this country, I’d like to know what it is.

In the 19th century landowners paid tax on their land. Today, so corrupt is our system of taxation, they actually receive subsidies for it. The rest of us, meanwhile, must pay council tax.

The largest landowners exploit a tax loophole. Land is passed from one generation to the next via the tax avoidance vehicle that is the trust. The rest of us must pay inheritance tax.

The complexity and inconsistency of our tax systems are to blame for so much wealth inequality. One group has the resources to find the loopholes and exploit them, the rest of us don’t: and so pay more on a proportional basis. Complexity allows there to be one rule for some and another for everybody else.

About the only way the person who starts out with nothing can improve his or her lot is through labour. And yet we tax labour constantly and heavily. The worker pays the vast majority of taxes: 40% of government revenue comes from income tax and national insurance, with another 20% from VAT.

The wealth of the super-rich does not derive from their labour, however. It derives from the appreciation in the value of their land, their houses, their stocks, their shares, their bonds, their fine art – what economists call their assets. These go untaxed, unless you sell. So most don’t.

If you want to redistribute wealth naturally, rather than via the moral minefield that is state re-allocation, the answer lies in changing the way we tax people.

‘The late duke may have been a canny businessman, but he did not invent anything new.’ A shooting party on the Duke of Westminster’s estate in the Trough of Bowland. Photograph: Don McPhee/The Guardian

Instead of taxing our labour – what we produce – why don’t we tax what we use? Instead of taxing the wealth that is earned, why don’t we tax the wealth that is unearned? I’m talking about land. Nobody made the land. Nature gave it to us. By building on it, or farming it, or mining it, you have improved it, but the land itself was always there. So let us look solely at the unimproved value of the land. This is easy to assess.

If you want the right to occupy a piece of land, and you want the government to protect your title to that land, then a rent should be paid to the community that reflects the value of that land, because it is the needs of the community which have given that land value. What I’m describing might sound extremely left wing, but the granddaddy of rightwing economists, Milton Friedman, described it as the, “least bad tax”: that is LVT – land value tax.

Who would pay the most if we hand land value tax in the UK? The Queen (she owns most of it), the Duke of Buccleuch, the Duke of Atholl, Captain Alwyne Farquharson, pension funds, the Forestry Commission, the Ministry of Defence and, of course, the new Duke of Westminster – or rather the Grosvenor Trust, which owns the land.

Remember the mantra: don’t tax labour, tax land

The late duke may have been a canny businessman, but he did not invent anything new, he did not bring some amazing new product or service to the world, which we all wanted to use. His ancestors benefited from the corn laws 200 years ago and the estates were built. Now planning laws are such that few can build anything new. The estate, which owns some of the most desirable land in London, was effectively handed a monopoly and the duke made good from the fact that so many people want to live and work in London.

There’s big money to be made in land banking but there is nothing creative about it. You are not bringing anything new to the world or improving it. It is simply exploiting the restrictive planning laws in this country that prevent progress. It is crony capitalism at its worst.

If you don’t want to pay land value tax, you don’t have to. This is a tax that is voluntary. You simply sell the land to someone who is prepared to.

The amounts of tax payable are clear. It’s an easy tax to administer. It doesn’t require 10 million words of tax code. And there need be no loopholes. The land is here – it is not in the Cayman Islands – and you are the owner.

The Green party actually has LVT in its manifesto, but it has it in addition to other taxes. LVT should replace other taxes.

Remember the mantra: don’t tax labour, tax land. Not only would it make for a much healthier, happier and more productive society, it would make for one in which wealth is more fairly distributed.

• Dominic Frisby is performing his show Let’s Talk About Tax at the Edinburgh Festival until 28 August