If you live in Bolton and earn more than £80,000, you’re rich compared with your neighbours. Really rich. The average pay for all workers is just £22,000 and one in four earns less than £15,000. And yes, you’re still rich among the country as a whole: only 5% of earners make more than £80,000 – about 1.6 million people.

That’s the context to the Question Time row in Bolton – with an audience member refusing to accept that a tax on those, like him, earning over £80,000 would hit only top earners. He was very wrong, but he’s not alone in misreading where he stands versus other earners. People tend to assume they’re closer to the middle than they actually are, whether they’re rich or poor.

Why is this? Because your earnings, like most things in life, are relative. If you’re a GP or a lawyer you may well be earning £80,000 (the typical salary for these jobs is £76,000 and £70,000 respectively). If you hang around with other GPs or lawyers, many are likely to earn more than you. So among your colleagues and peers you are a typical earner. You just need to remember that not everyone in Britain does those jobs.

But there are some other lessons from the indignation of the man in the BBC audience. While he seems to misunderstand his own privilege, there’s also a mismatch between whom Labour talk about paying more tax, and those who will actually do so. When talking about the rich, Labour refers to bankers, big business and the wealthy elite. These people aren’t the top 5%. Many of them aren’t even the top 1%. They’re more like the top 0.1%.

And it is these staggeringly high-income people who have pulled away from the rest of us this century. The result is that today someone just inside the top 5% club, on £80,000, may have much more common with your average worker than the super-rich elite in the 0.1% club. So maybe it’s no wonder that a well-off employee in Bolton isn’t going to love being lumped in with globetrotting plutocrats in Kensington.

The other context to the Question Time questioner’s anger is that you don’t just compare your salary with those of your peers and the rest of the country: you compare it with what it was one, five or 10 years ago – and what you expected it to be by now. And from this perspective, we’re all right to be angry. That’s because our pay packets have been a disaster zone for the past decade – we’ve experienced the longest pay downturn in more than two centuries.

A back street between two rows of terraced houses in Bolton, one of Britain’s most deprived towns. Photograph: Alamy

Victoria Road in Kensington, home to some of the richest people in the country. Photograph: Google Streetview

This is the big-picture backdrop to Britain that means, for rich and poor alike, the last decade has felt like a very tough time. Understanding this helps us to recognise why anger at the UK’s high levels of inequality has grown over the past decade, despite the actual level of inequality barely shifting since the 1980s.

What does this mean for policymakers and voters as we face an election? That we should think differently about how we tax the well-off, and the arguments we make for it.

First, taxing the top will need to go beyond incomes to tax wealth. You may need to earn only £80,000 to join the 5% club for earners, but that’s unlikely to ever be enough for you to join the top 5% club by wealth. To achieve that, you need housing or savings of almost £1m – which you’ll probably have to inherit or marry rather than earn. That’s the result of our stagnant incomes but soaring wealth of recent years. The result is huge wealth gaps that are also hugely under-taxed.

Second, those wanting to see large increases in spending, not just on the likes of NHS and schools but also on free broadband, will need broader arguments for higher taxes. They will have to embrace the shared sacrifice required to build a better country, not just the anger many of us feel about the very rich running away from the rest of us.

• Torsten Bell is director of the Resolution Foundation