The fallout from the Panama Papers has to be about more than David Cameron and criticising the wealthy. The leaked documents that lifted the lid on a teeming mass of slippery backhand deals that benefit the super-rich and disadvantage everyone else, and the revelations that our Prime Minister profited from offshore investments should be public knowledge. But there’s no point just bashing those with huge amounts of money: Labour are right to challenge the corrupt system that means there’s one rule for super-rich and another for everyone else.

Take Sir Alan Duncan’s recent outburst. In the throes of a heated Commons debate about tax evasion on Monday, he publicly attacked people who aren’t rich as “low achievers”. Now, there’s a colossal flaw in his argument: the people he refers to – the teachers, nurses, cleaners, farmers and small business owners – are the same ones who keep this country going. The reason the vast majority of them go home each month with a paltry pay packet is that we live in a society, crafted by people like Duncan, that doesn’t appreciate the work they do. Meanwhile there are plenty of mediocre men in positions of power or with offshore bank accounts bursting at the seams, because they just so happen to be born into a social structure that’s skewed in their favour.

But it’s relatively easy to attack Duncan – the man who claimed thousands of pounds for “grass cutting” and his incoherent argument, what’s harder is to challenge the society that produced these ideas. Because Duncan isn’t alone. The commonly accepted idea that people who claim benefits are lazy springs from this same logic.

It’s within this same warped world that tax evasion finds a comfortable spot for itself. The super-rich have crafted an economic structure where almost every roll of the dice benefits them, and the rest of us are left to try our luck. All the while we’re told they believe that meritocracy is alive and well. Or as Nickholas Shaxson puts it the offshore system “helps rich people, companies and countries stay on top, for no good economic or political reason. It’s the battleground of the rich versus the poor, you versus the corporations, the havens against the democracies – and in each battle, unless you’re very rich, you are losing.”

To start dismantling this financially unfair labyrinth you have to focus on the big picture. Clamping down on large-scale tax evasion can’t be boiled down to hating rich people – as Duncan suggested – or haranguing the Tories as the party of the posh and privileged. These are tired out lines that perpetuate class warfare and alienate the public at large. But Labour’s message should be this: the Conservatives are a party that is happy to rig the deck against the majority of people. They’re key cogs in a system that props up inequality at the expense of all but a very few.

Whenever there’s a clear act of wrongdoing that exposes the gross inequality in society, it’s always tempting to blame the person at fault. But then they become the story, when it’s about so much more. That’s exactly what could happen with the Panama Papers; we shouldn’t reduce the debate to one about wealthy individuals, where criticisms of tax evasion are dismissed as the politics of envy. We should be calling the large-scale tax evasion out for what it is: corruption.

That doesn’t mean dismissing transparency as unnecessary; it’s only right we’re privy to the tax affairs of people like Boris Johnson because he’s an elected representative, who could soon slither in to the cabinet, and has a huge say in our democracy. It matters if they’re using their political clout for their own gain. But, focussing on the individual wrongdoing alone won’t lead to change. People won’t be around forever but unless we make are our arguments clear and compelling, the system could be.