There are bad ideas, really bad ideas, and there’s abolishing inheritance tax. This is the new top priority announced by the Brexit party – one of only two policies it has beyond Brexit.

Scrapping inheritance tax is a very expensive pledge: it raised £5.3bn for the Treasury last year. And it’s particularly expensive when there’s a political consensus that we need to spend more – earlier this month a Conservative government set out proposals to increase the size of the state, at the price of binning their fiscal rules.

Of the 10 constituencies with the highest numbers of inheritance tax payers, all 10 voted remain

But just because something’s expensive doesn’t necessarily mean it’s a bad idea, not least if lots of people would benefit from it. Universal education and healthcare cost quite a lot after all. But universal this policy is not – the estates of only 24,500 of those who passed away in 2015-16 paid any inheritance tax and would have benefited from its abolition. That’s just 4% of all UK deaths. If three people benefited from each of those estates, even Boris Johnson’s very expensive and very regressive proposed giveaway to higher-rate taxpayers would benefit 60 times as many people each year.

Scrapping inheritance tax would give that small number of estates an average of £179,000 each. And even that average hides the scale of the giveaway to the super-rich. If we’d abolished inheritance tax in 2015-16, 70% of the benefit would have gone to those with estates of over £1m, and a huge 40% to just 528 estates worth over £2m. That’s 528 people getting an average tax cut of just over £1m each, at a time when child poverty is rising.

The geographic spread of the tax cut is also interesting. The Brexit party rightly notes that the north of England has been let down, in part by too narrow a focus on London (their other non-Brexit policy is a big infrastructure investment programme for the regions outside London). Given that perspective, this tax cut is an odd priority: London and south-east England would win again – these regions pay twice as much inheritance tax as anywhere else. It’s also noteworthy that of the 10 constituencies with the highest numbers of inheritance tax payers – Twickenham and Richmond Park top the list – all 10 voted remain.

So it’s a very regressive policy that exacerbates regional inequality. The lesson of the past 10 years doesn’t seem to be that those are desirable attributes in a policy.

But even if you don’t mind prioritising the rich or the south-east, there are other reasons for caution. Scrapping inheritance tax fundamentally fails to recognise one of the biggest trends shaping modern Britain – wealth has become a bigger deal.

Since the 1980s, household wealth has grown much faster than our incomes. The result is that it’s now very, very hard to become rich without making sure you had the right parents – or turned things around later by marrying someone who did.

That’s why inheritances have doubled over the past 20 years, and why we project that they are set to double again over the next 20. They will increasingly determine our lifetime living standards – shaping both who can live where and the kind of lives we live. Policymakers can’t and shouldn’t completely insulate us from the effects of wealth being passed down, but they shouldn’t make it worse. And in the long term, less inheritance tax for a few means higher taxes on income and consumption for everyone else.

Inheritance tax: 'seven-year rule' on gifts could be cut to five Read more

So why is the Brexit party proposing it? There is a logic: it’s a really unpopular tax – with some good reason. Any tax on dying is a tough sell: even though hardly anyone pays inheritance tax, many worry they will. The headline 40% tax rate means people fear losing up to half their estates, despite the actual average tax rate, even for the largest estates, being below 20%. And there are far too many loopholes that can be exploited by those with large estates and tax advisers.

So instead of scrapping the tax we should replace it with one paid by those who are lucky enough to receive an inheritance, rather than those unlucky enough to pass away (at present your estate pays inheritance tax based on how big it is, rather than recipients paying based on how much they receive). Combining this with removing or restricting the most abused exemptions would allow the headline tax rate to be cut. That’s how we prevent the biggest estates avoiding the tax and return a sense of fairness to the system.

Whether you’re for leave or remain, it’s crucial to recognise that fairer taxes on wealth are crucial to making a success of 21st-century Britain. Scrapping inheritance tax isn’t.

• Torsten Bell is director of the Resolution Foundation