Even if he hasn't yet debased the coinage, Bob Diamond has certainly done his bit to debase further the language of British public life. Confronted with a clear ruling that Barclays traders had lied and cheated in seeking to rig a key interest rate used to determine everything from mortgages to credit card bills, Diamond put his hands up and conceded that the traders' action had been "wholly inappropriate".

Inappropriate? Inappropriate is wearing a tie to a barbecue. Wholly inappropriate is burping during the wedding vows. Distorting for personal gain a rate that underpins contracts worth $350 trillion worldwide is rather more than "inappropriate".

Still, Diamond insisted he wanted to "demonstrate responsibility" for what had happened. There was a time when, if someone in a position of authority took responsibility for a grave error or worse, their next move was resignation: "The buck stops with me and so, with a heavy heart, I must …" Not now. How exactly does Diamond propose to demonstrate his responsibility? He and three colleagues have "volunteered to forgo any consideration for bonuses in 2012, recognising our responsibility as leaders of the organisation in which these events occurred". Imagine a footballer implicated in a match-fixing plot suggesting his punishment should be to forgo that season's pot of extra prize money – while remaining on his telephone-number salary. It would seem – what's the word? – inappropriate.

But those in the financial stratosphere apparently see matters of right and wrong, crime and punishment, rather differently to the rest of us. It was the same Bob Diamond who last year told MPs, "There was a period of remorse and apology for banks and I think that period needs to be over" – calling time on the era of bankers' penitence before most thought it had begun.

In his remarks both last year and this week, the Barclays chief revealed more than merely his own desire to move on. He also exposed the gaping absence of accountability in our financial system – and not only there. Indeed, a specific lack of accountability lies at the root of this latest scandal. The eyes of the non-expert glaze over, but the crucial fact about the Libor rate is that it is set daily not by the government or the Financial Services Authority but by the banks themselves, through their own trade body, the British Bankers' Association. Those recently appalled at the way the British press works – through self-regulation by the now-discredited Press Complaints Commission – might want to reserve some shock for the way that, when it comes to this vital component of global finance, the banks are quite literally a law unto themselves, beyond the criminal enforcement powers even of the FSA.

And make no mistake, it is the banks plural we are discussing, not just Barclays. Submissions from some 15 banks are used to calculate the benchmark Libor rate, making it all but a technical impossibility that a few rogue traders at Barclays alone could have bent it. On the contrary, the most incriminating email to surface on Wednesday – traders promising to celebrate their fiddling of the figures with a bottle of Bollinger – was from an outside bank to Barclays. The latter is surely only in the frame first because it co-operated early in return for a more lenient penalty, but HSBC, RBS and others are all mentioned in court papers. As the chancellor and others have signalled, this scandal is going to spread much wider.

And yet, what can be done? You'd think criminal prosecutions would be the obvious next step, but it's not so simple: Libor falls outside the FSA's remit. Yes, there's that £290m fine – though it's worth noting only £60m of that was imposed by the UK, the rest demanded by American authorities. What's more, that £290m is destined not for the public coffers but for the FSA, which will therefore need to levy less from the banks that fund it – including Barclays. So Barclays lose with one hand but are set to gain with the other. Above all, remember that that £290m is about a tenth of the £2.7bn bonus pool Barclays top dogs paid themselves in 2011. It's more than a slap in the wrist, but not much more.

It's quite a contrast with the severity of punishment meted out to those guilty of more visible crimes, starting with the 1,292 people jailed for their part in last summer's riots, including the man imprisoned for six months for stealing bottles of water worth £3.50. There was no question of the authorities lacking a proper remit then, nor did any rioter have the chance to tell a parliamentary committee it was time we all moved on.

Some will say that now, at long last, a reckoning is coming. Note the governor of the Bank of England's unsparing condemnation of the "deceitful manipulation" of the Libor rate along with today's revelation of yet another mis-selling scandal. Note too, the FSA's Adair Turner damning of a banking culture rotten with "cynicism and greed".

That sounds promising, a harbinger of serious reform. But imagine this was last summer, and King and Turner were speaking not of bankers but rioters. Imagine they confined themselves to condemning the underlying cultural malaise, the structural flaws, the excessive greed and consumerism – and promised to make changes that would, in a decade or two, prevent such problems occurring again. They would be denounced as limp-wristed liberals, too blind to see that the first step required was immediate and severe action against guilty individuals, especially the ringleaders.

So it should be with this latest banking scandal, and indeed the scandalous conduct that caused the global crash of 2008, a crime that remains unpunished. It's all very well to speak of changing banking culture and it certainly needs to change. If a judicial inquiry is necessary for the press, it's clearly necessary for the banks.

But that cannot be an excuse for more of the long-grass mentality that won't see the Vickers recommendations on splitting the banks fully implemented until 2018. Even long-term work begins with a first step – and the first step now should be enforcing the rules we have, then drafting new ones to catch and punish those who have done wrong.

So far the public have been remarkably restrained in their anger. But that anger is rising, and Britain's masters – politicians and bankers – would be wise not to push the people's patience too far.

Twitter: @j_freedland