‘Guys, you’ve got to hear this,” I said. I was sitting in front of my computer one day in July 2012, with one eye on a screen of share prices and the other on a live stream of the House of Commons Treasury select committee hearings. As the Barclays share price took a graceful swan dive, I pulled my headphones out of the socket and turned up the volume so everyone could hear. My colleagues left their terminals and came around to watch BBC Parliament with me.

It didn’t take long to realise what was happening. “Bob’s getting murdered,” someone said.

Bob Diamond, the swashbuckling chief executive of Barclays, had been called before the committee to explain exactly what his bank had been playing at in regards to the Libor rate-fixing scandal. The day before his appearance, he had made things very much worse by seeming to accuse the deputy governor of the Bank of England of ordering him to fiddle an important benchmark, then walking back the accusation as soon as it was challenged. He was trying to turn on his legendary charm in front of a committee of angry MPs, and it wasn’t working. On our trading floor, in Mayfair, calls were coming in from all over the City. Investors needed to know what was happening and whether the damage was reparable.

A couple of weeks later, the damage was done. The money was gone, Diamond was out of a job and the market, as it always does, had moved on. We were left asking ourselves: How did we get it so wrong?

At the time I was working for a French stockbroking firm, on the team responsible for the banking sector. I was the team’s regulation specialist. I had been aware of “the Libor affair”, and had written about it on several occasions during the previous months. My colleagues and I had assumed that it would be the typical kind of regulatory risk for the banks – a slap on the wrist, a few hundred million dollars of fines, no more than that.

The first puzzle was that, to start with, it looked like we were right. By the time it caught the attention of the mainstream media, the Libor scandal had reached what would usually be the end of the story – the announcement, on 27 June 2012, of a regulatory sanction. Barclays had admitted a set of facts, made undertakings not to do anything similar again, and agreed to pay fines of £59.5m to the UK’s Financial Services Authority, $200m to the US Commodity Futures Trading Commission and a further $160m to the US Department of Justice. That’s how these things are usually dealt with. If anything, it was considered quite a tough penalty.

But the Libor case marked the beginning of a new process for the regulators. As well as publishing their judgment, they gave a long summary of the evidence and reasoning that led to their decision. In the case of the Libor fines, the majority of that evidence took the form of transcripts of emails and Bloomberg chat. Bloomberg’s trading terminals – the $50,000-a-year news and financial-data servers that every trader uses – have an instant-messaging function in addition to supplying prices and transmitting news. Financial market professionals are vastly more addicted to this chat than tween girls are to Instagram, and many of them failed to realise that if you discussed illegal activity on this medium, you were making things easy for the authorities.

The transcripts left no room for doubt.

Trader C: “The big day [has] arrived … My NYK are screaming at me about an unchanged 3m libor. As always, any help wd be greatly appreciated. What do you think you’ll go for 3m?”

Submitter: “I am going 90 altho 91 is what I should be posting.”

Trader C: “[…] when I retire and write a book about this business your name will be written in golden letters […]”.

Submitter: “I would prefer this [to] not be in any book!”

Perhaps it’s unfair to judge the Libor conspirators on their chat records; few of the journalists who covered the story would like to see their own Twitter direct-message history paraded in front of an angry public. Trading, for all its bluster, is basically a service industry, and there is no service industry anywhere in the world whose employees don’t blow off steam by acting out or insulting the customers behind their backs. But traders tend to have more than the usual level of self-confidence, bordering on arrogance. And in a general climate in which the public was both unhappy with the banking industry and unimpressed with casual banter about ostentatious displays of wealth, the Libor transcripts appeared crass beyond belief. Every single popular stereotype about traders was confirmed. An abstruse and technical set of regulatory breaches suddenly became a morality play, a story of swaggering villains who fixed the market as if it was a horse race. The politicians could hardly have failed to get involved.

It is not a pleasant thing to see your industry subjected to criticism that is at once overheated, ill-informed and entirely justified. In 2012, the financial sector finally got the kind of enemies it deserved. The popular version of events might have been oversimplified and wrong in lots of technical detail, but in the broad sweep, it was right. The nuanced and technical version of events which the specialists obsessed over might have been right on the detail, but it missed one utterly crucial point: a massive crime of dishonesty had taken place. There was a word for what had happened, and that word was fraud. For a period of months, it seemed to me as if the more you knew about the Libor scandal, the less you understood it.

That’s how we got it so wrong. We were looking for incidental breaches of technical regulations, not systematic crime. And the thing is, that’s normal. The nature of fraud is that it works outside your field of vision, subverting the normal checks and balances so that the world changes while the picture stays the same. People in financial markets have been missing the wood for the trees for as long as there have been markets.

Some places in the world are what they call “low-trust societies”. The political institutions are fragile and corrupt, business practices are dodgy, debts are rarely repaid and people rightly fear being ripped off on any transaction. In the “high-trust societies”, conversely, businesses are honest, laws are fair and consistently enforced, and the majority of people can go about their day in the knowledge that the overall level of integrity in economic life is very high. With that in mind, and given what we know about the following two countries, why is it that the Canadian financial sector is so fraud-ridden that Joe Queenan, writing in Forbes magazine in 1989, nicknamed Vancouver the “Scam Capital of the World”, while shipowners in Greece will regularly do multimillion-dollar deals on a handshake?

We might call this the “Canadian paradox”. There are different kinds of dishonesty in the world. The most profitable kind is commercial fraud, and commercial fraud is parasitical on the overall health of the business sector on which it preys. It is much more difficult to be a fraudster in a society in which people only do business with relatives, or where commerce is based on family networks going back centuries. It is much easier to carry out a securities fraud in a market where dishonesty is the rare exception rather than the everyday rule.

Traders at Bloomberg terminals on the floor of the New York stock exchange, 2013. Photograph: Brendan McDermid / Reuters/REUTERS

The existence of the Canadian paradox suggests that there is a specifically economic dimension to a certain kind of crime of dishonesty. Trust – particularly between complete strangers, with no interactions beside relatively anonymous market transactions – is the basis of the modern industrial economy. And the story of the development of the modern economy is in large part the story of the invention and improvement of technologies and institutions for managing that trust.

And as industrial society develops, it becomes easier to be a victim. In The Wealth of Nations, Adam Smith described how prosperity derived from the division of labour – the 18 distinct operations that went into the manufacture of a pin, for example. While this was going on, the modern world also saw a growing division of trust. The more a society benefits from the division of labour in checking up on things, the further you can go into a con game before you realise that you’re in one. In the case of several dealers in the Libor market, by the time anyone realised something was crooked, they were several billions of dollars in over their heads.

In hindsight, the Libor system was always a shoddy piece of work. Some not-very-well-paid clerks from the British Bankers’ Association would call up a few dozen banks and ask: “If you were to borrow, say, a million dollars in [a given currency] for a 30-day deposit, what would you expect to pay?” A deposit, in this context, is a short-term loan from one bank to another. Due to customers’ inconvenient habit of borrowing from one bank and putting the money in an account at another, banks are constantly left with either surplus customer deposits, or a shortage of funds. The “London inter-bank offered-rate” (Libor) market is where they sort this out by borrowing from and lending to each other, at the “offered rate” of interest.

Once they had their answers, the clerks would throw away the highest and lowest outliers and calculate the average of the rest, which would be recorded as “30-day Libor” for that currency. The process would be repeated for three-month loans, six-month loans and any other periods of interest, and the rates would be published. You would then have a little table recording the state of the market on that day – you could decide which currency you wanted to borrow in, and how long you wanted the use of the money, and the Libor panel would give you a good sense of what high-quality banks were paying to do the same.

Compared with the amount of time and effort that goes into the systems for nearly everything else that banks do, not very much trouble was taken over this process. Other markets rose and fell, stock exchanges mutated and were taken over by super-fast robots, but the Libor rate for the day was still determined by a process that could be termed “a quick ring-around”. Nobody noticed until it was too late that hundreds of trillions of dollars of the world economy rested on a number compiled by the few dozen people in the world with the greatest incentive to fiddle it.

It started to fall apart with the onset of the global financial crisis in 2007, and all the more so after the collapse of Lehman Brothers in 2008, when banks were so scared that they effectively stopped lending to each other. Although the market was completely frozen, the daily Libor ring-around still took place, and banks still gave, almost entirely speculatively, answers to the question “If you were to borrow a reasonable sum, what would you expect to pay?”

But the daily quotes were published, and that meant everyone could see what everyone else was saying about their funding costs. And one of the telltale signs that a bank in trouble is when its funding costs start to rise. If your Libor submission is taken as an indicator of whether you’re in trouble or not, you really don’t want to be the highest number on the daily list. Naturally, then, quite a few banks started using the Libor submission process as a form of false advertising, putting in a lowballed quote in order to make it look like they were still obtaining money easily when, in fact, they could hardly borrow at all. And so it came to pass that several banks created internal message trails saying, in effect, “Dear Lowly Employee, for the benefit of the bank and its shareholders, please start submitting a lower Libor quote, signed Senior Executive”. This turned out to be a silly thing to do.

All this was known at the time. There was an article in the Wall Street Journal about it. I used to prepare PowerPoint slides with charts on them that had gaps for the year 2008 because the data was “somewhat hypothetical”. Even earlier, in late 2007, the Bank of England held a “liaison group” meeting so that representatives from the banks could discuss the issue of Libor reporting. What nobody seemed to realise is that an ongoing fraud was being committed. There was a conspiracy to tell a lie (to the Libor phone panel, about a bank’s true cost of funding) in order to induce someone to enter into a bargain at a disadvantage to themselves. The general public caught on to all this a lot quicker than the experts did, which put the last nail in the coffin of the already weakened trust in the financial system. You could make a case that a lot of the populist politics of the subsequent decade can be traced back to the Libor affair.

Libor teaches us a valuable lesson about commercial fraud – that unlike other crimes, it has a problem of denial as well as one of detection. There are very few other criminal acts where the victim not only consents to the criminal act, but voluntarily transfers the money or valuable goods to the criminal. And the hierarchies, status distinctions and networks that make up a modern economy also create powerful psychological barriers against seeing fraud when it is happening. White-collar crime is partly defined by the kind of person who commits it: a person of high status in the community, the kind of person who is always given the benefit of the doubt.

In popular culture, the fraudster is the “confidence man”, somewhere between a stage magician and the trickster gods of mythology. In films such as The Sting and Dirty Rotten Scoundrels, they are master psychologists, exploiting the greed and myopia of their victims, and creating a world of illusion. People like this do exist (albeit rarely). But they are not typical of white-collar criminals.

The interesting questions are never about individual psychology. There are plenty of larger-than-life characters. But there are also plenty of people like Enron’s Jeff Skilling and Baring’s Nick Leeson: aggressively dull clerks and managers whose only interest derives from the disasters they caused. And even for the real craftsmen, the actual work is, of necessity, incredibly prosaic.

The way most white-collar crime works is by manipulating institutional psychology. That means creating something that looks as much as possible like a normal set of transactions. The drama comes later, when it all unwinds.

Fraudsters don’t play on moral weaknesses, greed or fear; they play on weaknesses in the system of checks and balances – the audit processes that are meant to supplement an overall environment of trust. One point that comes up again and again when looking at famous and large-scale frauds is that, in many cases, everything could have been brought to a halt at a very early stage if anyone had taken care to confirm all the facts. But nobody does confirm all the facts. There are just too bloody many of them. Even after the financial rubble has settled and the arrests been made, this is a huge problem.

Jeffrey Skilling and Sherron Watkins of Enron at a Senate commerce committee hearing in 2002. Photograph: Ron Edmonds/AP

It is a commonplace of law enforcement that commercial frauds are difficult to prosecute. In many countries, proposals have been made, and sometimes passed into law, to remove juries from complex fraud trials, or to move the task of dealing with them out of the criminal justice system and into regulatory or other non-judicial processes. Such moves are understandable. There is a need to be seen to get prosecutions and to maintain confidence in the whole system. However, taking the opinions of the general public out of the question seems to me to be a counsel of despair.

When analysed properly, there isn’t much that is truly difficult about the proverbial “complex fraud trial”. The underlying crime is often surprisingly crude: someone did something dishonest and enriched themselves at the expense of others. What makes white-collar trials so arduous for jurors is really their length, and the amount of detail that needs to be brought for a successful conviction. Such trials are not long and detailed because there is anything difficult to understand. They are long and difficult because so many liars are involved, and when a case has a lot of liars, it takes time and evidence to establish that they are lying.

This state of affairs is actually quite uncommon in the criminal justice system. Most trials only have a couple of liars in the witness box, and the question is a simple one of whether the accused did it or not. In a fraud trial, rather than denying responsibility for the actions involved, the defendant is often insisting that no crime was committed at all, that there is an innocent interpretation for everything.

In January this year, the construction giant Carillion collapsed. Although they had issued a profits warning last summer, they continued to land government contracts. It was assumed that, since they had been audited by KPMG, one of the big-four accounting firms, any serious problems would have been spotted.

At the time of writing, nobody has been prosecuted over the collapse of Carillion. Maybe nobody will and maybe nobody should. It’s possible, after all, for a big firm to go bust, even really suddenly, without it being a result of anything culpable. But the accounting looks weird – at the very least, they seem to have recognised revenue a long time before it actually arrived. It’s not surprising that the accounting standards bodies are asking some questions. So are the Treasury select committee: one MP told a partner at KPMG that “I would not hire you to do an audit of the contents of my fridge.”

In general, cases of major fraud should have been prevented by auditors, whose specific job it is to review every set of accounts as a neutral outside party, and certify that they are a true and fair view of the business. But they don’t always do this. Why not? The answer is simple: some auditors are willing to bend the rules, and some are too easily fooled. And whatever reforms are made to the accounting standards and to the rules governing the profession, the same problems have cropped up again and again.

First, there is the problem that the vast majority of auditors are both honest and competent. This is a good thing, of course, but the bad thing about it is that it means that most people have never met a crooked or incompetent auditor, and therefore have no real understanding that such people exist.

To find a really bad guy at a big-four accountancy firm, you have to be quite unlucky (or quite lucky if that was what you were looking for). But as a crooked manager of a company, churning around your auditors until you find a bad ’un is exactly what you do – and when you find one, you hang on to them. This means that the bad auditors are gravitationally drawn into auditing the bad companies, while the majority of the profession has an unrepresentative view of how likely that could be.

Second, there is the problem that even if an auditor is both honest and competent, he has to have a spine, or he might as well not be. Fraudsters can be both persistent and overbearing, and not all the people who went into accountancy firms out of university did so because they were commanding, alpha-type personalities.

Added to this, fraudsters are really keen on going over auditors’ heads and complaining to their bosses at the accounting firm, claiming that the auditor is being unhelpful and bureaucratic, not allowing the CEO to use his legitimate judgment in presenting the results of his own business.

Partly because auditors are often awful stick-in-the-muds and arse-coverers, and partly because auditing is a surprisingly competitive and unprofitable business that is typically used as a loss-leader to sell more remunerative consulting and IT work, you can’t assume that the auditor’s boss will support their employee, even though the employee is the one placing their signature (and the reputation of the whole practice) on the set of accounts. As with several other patterns of behaviour that tend to generate frauds, the dynamic by which a difficult audit partner gets overruled or removed happens so often, and reproduces itself so exactly, that it must reflect a fairly deep and ubiquitous incentive problem that will be very difficult to remove.

By way of a second line of defence, investors and brokerage firms often employ their own “analysts” to critically read sets of published accounts. The analyst is meant to be an industry expert, with enough financial training to read company accounts and to carry out valuations of companies and other assets. Although their primary job is to identify profitable opportunities in securities trading – shares or bonds that are either very undervalued or very overvalued – it would surely seem to be the case that part of this job would involve the identification of companies that are very overvalued because they are frauds.

Well, sometimes it works. A set of fraudulent accounts will often generate “tells”. In particular, fraudsters in a hurry, or with limited ability to browbeat the auditors, will not be able to fake the balance sheet to match the way they have faked the profits. Inflated sales might show up as having been carried out without need for inventories, and without any trace of the cash they should have generated. Analysts are also often good at spotting practices such as “channel stuffing”, when a company (usually one with a highly motivated and target-oriented sales force) sells a lot of product to wholesalers and intermediaries towards the end of the quarter, booking sales and moving inventory off its books. This makes growth look good in the short term, at the expense of future sales.

Often, an honest auditor who has buckled under pressure will include a cryptic-looking passage of legalese, buried in the notes to the accounts, explaining what accounting treatment has been used, and hoping that someone will read it and understand that the significance of this note is that all of the headline numbers are fake. Nearly all of the fraudulent accounting policies adopted by Enron could have been deduced from its public filings if you knew where to look.

More common is the situation that prevailed in the period immediately preceding the global financial crisis.Analysts occasionally noticed that some things didn’t add up, and said so, and one or two of them wrote reports that, if taken seriously, could have been seen as prescient warnings. The problem is that spotting frauds is difficult and, for the majority of investors, not worth expending the effort on. That means it is not worth it for most analysts, either. Frauds are rare. Frauds that can be spotted by careful analysis are even rarer. And frauds that are also large enough to offer serious rewards for betting against them come along roughly once every business cycle, in waves.

Analysts are also subject to very similar pressures to those that cause auditors to compromise their principles. Anyone accusing a company publicly of being a fraud is taking a big risk, and can expect significant retaliation. It is well to remember that frauds generally look like very successful companies, and there are sound accounting reasons for this. It is not just that once you have decided to fiddle the accounts you might as well make them look great rather than mediocre.

If you are extracting cash fraudulently, you usually need to be growing the fake earnings at a higher rate. So people who are correctly identifying frauds can often look like they are jealously attacking success. Frauds also tend to carry out lots of financial transactions and pay large commissions to investment banks, all the while making investors believe they are rich. The psychological barriers against questioning a successful CEO are not quite as powerful as those against questioning the honesty of a doctor or lawyer, but they are substantial.

And finally, most analysts’ opinions are not read. A fraudster does not have to fool everyone; he just needs to fool enough people to get his money.

If you are looking to the financial system to protect investors, you are going to end up being disappointed. But this is inevitable. Investors don’t want to be protected from fraud; they want to invest. Since the invention of stock markets, there has been surprisingly little correlation between the amount of fraud in a market and the return to investors. It’s been credibly estimated that in the Victorian era, one in six companies floated on the London Stock Exchange was a fraud. But people got rich. It’s the Canadian paradox. Although in the short term, you save your money by checking everything out, in the long term, success goes to those who trust.

Main image: Getty/Guardian Design

Adapted from Lying for Money: How Fraud Makes the World Go Round by Dan Davies, published by Profile on 5 July, and available at guardianbookshop.com

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.