Maybe it’s the consequence of having a week off and breathing fresh country air, but by the end of my annual leave on Sunday I was beginning to feel a teeny bit sorry for David Cameron.

That, I should add, is a relatively unsettling emotion for me. Still, having watched the Panama Papers scandal unfold from afar I couldn’t help but feel that the media focus on the Prime Minister might have been a little overblown.

He has, as everyone accepts, done nothing illegal. Investing in an offshore unit trust to the tune of a few tens of thousands of pounds may be tax efficient because the trust itself was liable for non-UK corporation duties, but any number of pension funds invest in these vehicles too. As for the gift from his mother, however generous, it is undeniably the kind of thing that anyone with considerable wealth would presumably be likely – and allowed – to do. The Camerons say it was not, in any case, done primarily to limit their inheritance tax liabilities.

Whether Cameron is a hypocrite depends on your point of view. He has talked the talk on tax avoidance, so perhaps might reasonably have been expected to be beyond even moral reproach in his own tax planning. Then again, if anyone who has raged against Cameron’s investments has themselves ever paid cash in hand to a plumber, for instance, the same charge of hypocrisy must surely apply.

And really, if the Prime Minister is a hypocrite, then at least he’s a hypocrite who has taken some steps to make the UK a more transparent place insofar as our tax system is concerned. The problem, as revealed by the Panama Papers, is just how limited those improvements can be for as long as they don’t extend to offshore zones, including British Overseas Territories (BOTs) and Crown Dependencies.

10 of the biggest tax havens in the world Show all 10 1 /10 10 of the biggest tax havens in the world 10 of the biggest tax havens in the world Luxembourg There are an estimated £2.5 trillion shares of mutual funds registered in the Grand Duchy, £1 trillion of which cannot be traced to an owner 10 of the biggest tax havens in the world Cayman Islands The Cayman Islands contain 6% of the world's total banking assets, but just 0.000008% of its population 10 of the biggest tax havens in the world Isle of Man David Cameron has said the Isle of Man, where there is no corporation, capital gains or inheritance tax, should not be considered a tax haven 10 of the biggest tax havens in the world Jersey There are over £3.5 billion assets per square mile on the self-governing Channel Island 10 of the biggest tax havens in the world Ireland Ireland made headlines last year when it emerged Apple was registered in the country in order to dodge over £40bn in taxes 10 of the biggest tax havens in the world Mauritius The Mauritian government notionally charges corporation tax, but companies can easily make this back through generous tax credits for foreign businesses 10 of the biggest tax havens in the world Bermuda Google holds more than £30bn in offshore cash reserves, primarily via Bermuda 10 of the biggest tax havens in the world Monaco A popular domicile for super-rich private individuals, Monaco has the most expensive property in the world. £1 million will buy just 225 square feet 10 of the biggest tax havens in the world Switzerland Switzerland has such secretive banking laws that it took until the 1990s to secure the release of Nazi cash reserves 10 of the biggest tax havens in the world Bahamas David Cameron's father ran an offshore fund which hired Bahamas residents to complete paperwork, thus dodging British tax bills

This goes to the heart of why I think the spotlight should swerve away from Cameron for a moment – not because transparency over his own finances is an intrinsically bad thing (although one wonders where it will end) but because there are bigger issues at stake. Of course he is a useful symbol, but of what exactly? Well-off people planning their tax arrangements carefully? Well fine, and the merits of his actions can be debated. But focusing on the Prime Minister’s tax affairs runs the risk that we understate the degree to which other, genuinely shady, people use offshore bank accounts and shell companies to deliberately obscure trails of dirty money and launder vast sums so they can be accessed by concealed beneficiaries.

It is this interplay between the world of offshore finance and secretive, global criminality which should truly alarm us, and not least because of the role played by BOTs in enabling it to happen.

Indeed, for all that Mossack Fonseca declares itself entirely above board, the role of major offshore law firms – run, of course, by very wealthy lawyers enjoying the advantages of living in tax havens – and the way they help individuals set up shell operations is ripe for scrutiny. Journalism, the Church, politicians and bankers have all come under the microscope; the turn of the legal profession’s upper commercial echelons seems never to arrive.

Nobody would dispute that Cameron’s handling of last week’s revelations was dreadful and that itself inevitably gave rise to suspicion. But many of his most trenchant critics are motivated by politics as much as, if not more than, a desire to clean up offshore trading. Opponents on the left and Brexiteers on the right – including in the media – have found common cause.

If the pressure on the Prime Minister results in policy changes that improve offshore transparency, then perhaps it is worth it. Yet, as things stand, it feels as if the major endeavour is simply to uncover details about his own and his family’s wealth.

It may make for a simpler and more tangible narrative, but by holding the PM to account are we letting others, whose evils are much greater, slip off the hook?