Adam Smith would be the perfect expert witness in the case being brought by Guardian Care Homes (nothing to do with this newspaper) against Barclays bank, which has its first hearing in the high court on Monday.

Back in 1776, the sage of Kirkcaldy noted: "People of the same trade seldom meet together even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices."

The narrow issue is whether Guardian Care Homes was mis-sold financial instruments, known as swaps, that were marketed by banks as a way to protect businesses and public bodies against costly movements in interest rates.

But in essence the action is a test case that could have severe consequences for the banks. The charge against Barclays is that the £12m the company claims it lost was the result of manipulation of the Libor (London interbank offered rate), used to price all sorts of financial products from student loans to mortgages and borrowing costs on credit cards.

Libor is a strange creature, but the way it has been operated fits Smith's dictum to the letter. It is not like the bank rate, set monthly by the Bank of England's monetary policy committee and which stands at 0.5%. Nor is it akin to the interest rate paid on the gilts the government issues to finance its debt, which is fixed at auction and established by the myriad daily trades in the bond market.

Instead, Libor is set by a small number of big banks, who submit a rate each day at which they think they might be able to borrow rather than the rates at which they can actually borrow. After the highest and lowest submissions are trimmed, an average of the rest of the quotes becomes that day's Libor rate.

Getting the right Libor rate matters. Banks should not be able to fiddle the rate to screw their customers. Nor should they be able to submit misleading quotes in order to disguise just how bad a financial position they are in.

Imagine, for example, that the UK authorities had known in the first half of 2007 that some banks were paying a lot more to fund themselves than they were letting on. Alarm bells would have rung earlier and action might have taken that would have saved the taxpayer a packet.

Finally, there's the little matter of the financial derivatives that are priced against the rate. At a conservative estimate, outstanding derivatives contracts are worth $600tn (£372tn) – around 10 times the annual output of the global economy. It goes without saying that these complex financial instruments should be priced correctly.

The question, therefore, is whether it is sensible to leave such an important interest rate in the hands of a cabal of banks. Is it sufficient to put down the scandals exposed this year to a handful of rogue traders, poor internal management and inadequately policed regulatory safeguards, or is there a deeper malaise?

Enter poacher-turned-gamekeeper Alexis Stenfors, who received a five-year ban from working in the City after costing his employer Merrill Lynch $100m by concealing losses on his trading account. Stenfors is now an academic at the London School for Oriental and African Studies where he has just published two papers on Libor fixing.

Some will say that, as a former rogue trader, anything he says has to be taken with a pinch of salt. Yet, he was a trader for 15 years before his fall and he knows first-hand how the markets operate.

Put simply, the argument in the papers, available at researchonmoneyandfinance.org, is that there is a systemic problem with Libor. Although the banks would like us to believe that the scandal is down to a few bad apples, that is not the case.

The first paper analyses how banks have the means, the opportunities and the incentives to rig Libor in a way that is beneficial to themselves. They have the means because they collectively set the rate. They have the opportunities because the rates they submit have little or no bearing on those at which they actually trade. They have two big incentives to "game" the rate. First, they can enhance the value of their derivatives portfolios by manipulating Libor. Second, they can avoid being stigmatised as a "bank in trouble" by putting in a low rate that bears no relation to what they are actually paying for their finance.

In his paper, Stenfors conducts a number of different games, in some of which the banks collude with one another and in some of which they don't. In all the different scenarios, the outcome is a Libor rate that differs from the true funding costs of the banks. He also notes that the trimming process for removing the highest and lowest bids is ineffective and that bringing in new rules and constraints to enhance transparency provides disappointing results.

"Banks are given the chance to influence the Libor in a direction that is beneficial to them – stemming from the exclusive privilege to be able to play the game, in other words to participate in the Libor fixing process," he concludes.

The second paper takes Keynes's beauty contest theory as its starting point. Investors, he said, were like members of the public asked to choose the prettiest contestant; they would often not choose the one they actually liked best but rather the one they thought fellow judges would collectively pick.

The same principle, Stenfors says, applies to Libor. Banks are strongly influenced when they put in their daily Libor quote by what they think every other bank is doing. "Honest" banks have an incentive to act dishonestly because they fear that if they don't play the same game as all the other banks they will pay a financial cost.

While the papers are theoretical and offer no policy suggestions, there are certainly some important conclusions that can be drawn from them.

Clearly, Libor fixing is a case where self-regulation has proved to be useless. This might not matter much if the potential costs of getting Libor wrong were minor. But they are not: they are potentially astronomic, which perhaps explains why the authorities on both sides of the Atlantic have responded quickly to the scandal.

US financial watchdogs last week added nine more banks to their investigation into whether Libor has been rigged by the world's big banks. Barclays, Lloyds, RBS and HSBC are all under the spotlight.

The UK government has said it will accept the findings of Martin Wheatley's report into Libor. These include statutory regulation, making misleading Libor quotations a criminal offence, and giving the Financial Services Authority powers to force participating banks to abide by a new code of conduct. But if Stenfors is right and Libor is structurally flawed, reform will not be enough. This "conspiracy against the public" requires the Bank of England to design, run and police an entirely new system.