In our special briefing, we look at how near Wall Street came to systemic collapse this week—and how the financial system will change as a result. We start with how financiers—and their critics—have laboured under a delusion

“A COMPANY for carrying out an undertaking of great advantage, but nobody to know what it is.” This lure for the South Sea Company, published in 1720, has a whiff of the 21st century about it. Modern finance has promised miracles, seduced the brilliant and the greedy—and wrought destruction. Alan Greenspan, formerly chairman of the Federal Reserve, said in 2005 that “increasingly complex financial instruments have contributed to the development of a far more flexible, efficient, and hence resilient financial system than the one that existed just a quarter-century ago.” Tell that to Bear Stearns, Wall Street's fifth-largest investment bank, the most spectacular corporate casualty so far of the credit crisis.

For the critics of modern finance, Bear's swift end on March 16th was the inevitable consequence of the laissez-faire philosophy that allowed financial services to innovate and spread almost unchecked. This has created a complex, interdependent system prone to conflicts of interest. Fraud has been rampant in the sale of subprime mortgages. Spurred by pay that was geared to short-term gains, bankers and fund managers stand accused of pocketing bonuses with no thought for the longer-term consequences of what they were doing. Their gambling has been fed by the knowledge that, if disaster struck, someone else—borrowers, investors, taxpayers—would end up bearing at least some of the losses.

Since the era of frock coats and buckled shoes, finance has been knocked back by booms and busts every ten years or so. But the past decade has been plagued by them. It has been pocked by the Asian crisis, the debacle at Long-Term Capital Management, a super-brainy hedge fund, the dotcom crash and now what you might call the first crisis of securitisation. If the critics are right and something in finance is broken, then there will be pressure to reregulate, to return to what Alistair Darling, Britain's chancellor of the exchequer, calls “good old-fashioned banking”. But are the critics right? What really went wrong with finance? And how can it be fixed?

Happy days

The seeds of today's disaster were sown in the 1980s, when financial services began a pattern of growth that may only now have come to an end. In a recent study Martin Barnes of BCA Research, a Canadian economic-research firm, traces the rise of the American financial-services industry's share of total corporate profits, from 10% in the early 1980s to 40% at its peak last year (see chart 1). Its share of stockmarket value grew from 6% to 19%. These proportions look all the more striking—even unsustainable—when you note that financial services account for only 15% of corporate America's gross value added and a mere 5% of private-sector jobs.

At first this growth was built on the solid foundations of rising asset prices. The 18 years to 2000 witnessed an unparalleled bull market for shares and bonds. As the world's central banks tamed inflation, interest rates fell and asset prices rose (see chart 2). Corporate restructuring, wage competition and a revolution in information technology boosted profits. A typical portfolio of shares, bonds and cash gave real annual yields of over 14%, calculates Mr Barnes, almost four times the norm of earlier decades. Financial-service firms made hay. The number of equity mutual funds in America rose more than fourfold.

But something changed in 2001, when the dotcom bubble burst. America's GDP growth since then has been weaker than in any cycle since the 1950s, barring the double-dip recovery in 1980-81. Stephen King and Ian Morris of HSBC point out that growth in consumer spending, total investment and exports in this cycle has been correspondingly feeble.

Yet, like Wile E. Coyote running over the edge of a cliff, financial services kept on going. A service industry that, in effect, exists to help people write, trade and manage financial claims on future cashflows raced ahead of the real economy, even as the ground beneath it fell away.

The industry has defied gravity by using debt, securitisation and proprietary trading to boost fee income and profits. Investors hungry for yield have willingly gone along. Since 2000, according to BCA, the value of assets held in hedge funds, with their high fees and higher leverage, has quintupled. In addition, the industry has combined computing power and leverage to create a burst of innovation. The value of outstanding credit-default swaps, for instance, has climbed to a staggering $45 trillion. In 1980 financial-sector debt was only a tenth of the size of non-financial debt. Now it is half as big.

This process has turned investment banks into debt machines that trade heavily on their own accounts. Goldman Sachs is using about $40 billion of equity as the foundation for $1.1 trillion of assets. At Merrill Lynch, the most leveraged, $1 trillion of assets is teetering on around $30 billion of equity. In rising markets, gearing like that creates stellar returns on equity. When markets are in peril, a small fall in asset values can wipe shareholders out.

The banks' course was made possible by cheap money, facilitated in turn by low consumer-price inflation. In more regulated times, credit controls or the gold standard restricted the creation of credit. But recently central banks have in effect conspired with the banks' urge to earn fees and use leverage. The resulting glut of liquidity and financial firms' thirst for yield led eventually to the ill-starred boom in American subprime mortgages.

The dance of debt

The tendency for financial services to run right over the cliff is accentuated by financial assets' habit of growing during booms. By lodging their extra assets as collateral, the intermediaries can put them to work and borrow more. Tobias Adrian, of the Federal Reserve Bank of New York, and Hyun Song Shin, of Princeton University, have shown that since the 1970s, debts have grown faster than assets during booms. This pro-cyclical leverage can feed on itself. If financial groups use the borrowed money to buy more of the sorts of securities they lodged as collateral, then the prices of those securities will go up. That, in turn, enables them to raise more debt and buy more securities.

Indeed, their shareholders would punish them if they sat out the next round—as Chuck Prince let slip only weeks before the crisis struck, when he said that Citigroup, the bank he then headed, was “still dancing”. Mr Prince has been ridiculed for his lack of foresight. In fact, he was guilty of blurting out finance's embarrassing secret: that he was trapped in a dance he could not quit. As, in fact, was everyone else.

Sooner or later, though, the music stops. And when it does, the very mechanisms that create abundant credit will also destroy it. Most things attract buyers when the price falls. But not necessarily securities. Because financial intermediaries need to limit their leverage in a falling market, they sell assets (again, the system is pro-cyclical). That lowers the prices of securities, which puts further strain on balance sheets leading to further sales. And so the screw turns until those without leverage will buy.

You do not need bankers to be poorly monitored or over-incentivised for such cycles to work: finance knew booms and manias long before deposit insurance, bank rescues or bonuses. And, human nature being what it is, Jérôme Kerviel, who lost Société Générale a fortune, and the staff of various loss-making, state-owned, German Landesbanks did not need huge pay to lose huge sums. The desire to show that you are a match for the star trader next door, or the bank in the next town, will do.

Yet pay—or at least bad management—probably made this crisis worse. Trades determine bonuses at the end of the year, even though their real value may not become clear until later. Earlier this month a group of financial supervisors reported how managers at the banks worst hit by the crisis had failed to oversee traders or take a broad view of risk across their firms. Perhaps, with proper incentives, managers would have done better.

Alan Johnson, a consultant who designs pay packages for Wall Street, predicts that in future senior executives will face the prospect of some of their bonuses being contingent on the bank's performance over several years. Yet to the extent that many senior bankers are paid in shares they cannot immediately sell, they already are. And to the extent that Bear Stearns's employees owned one-third of the firm, they already looked to the longer term.

If altering pay cannot stop manias, can regulation? The criticism that this crisis is the product of the deregulation of finance misses an important point. The worst excesses in the securitisation mess are encrusted precisely where regulation sought to protect banks and investors from the dangers of untrammelled credit growth. That is because regulations offer not just protection, but also clever ways to make money by getting around them.

Existing rules on capital adequacy require banks to put some capital aside for each asset. If the market leads to losses, the chances are they will have enough capital to cope. Yet this rule sets up a perverse incentive to create structures free of the capital burden—such as credits that last 364 days, and hence do not count as “permanent”. The hundreds of billions of dollars in the shadow banking system—the notorious SIVs and conduits that have caused the banks so much pain—have been warehoused there to get round the rules. Spain's banking regulator prudently said that such vehicles could be created, but only if the banks put capital aside. So far the country has escaped the damage seen elsewhere. When reformed capital-adequacy rules are introduced, this is an area that will need to be monitored rigorously.

It is the same with rating agencies, the whipping boys of the crisis. Most bonds used to be issued by companies, and to judge something AAA was straightforward. Perhaps back then it made sense for some investors, such as pension funds, to be obliged to buy top-rated bonds. But this rule created a boundary between AAA and other bonds that was ripe for gaming. Clever people, abetted by the rating agencies, set out to pass off poor credit as AAA, because they stood to make a lot of money. And they did. For a while.

The financial industry is likely to stagnate or shrink in the next few years. That is partly because the last phase of its growth was founded on unsustainable leverage, and partly because the value of the underlying equities and bonds is unlikely to grow as it did in the 1980s and 1990s. If finance is foolishly reregulated, it will fare even worse.

And what of all the clever and misused wizardry of modern finance? Mr Greenspan was half right. Financial engineering can indeed spread risk and help the system work better. Like junk bonds, reviled at the end of 1980s, securitisation will rebound, tamed and better understood—and smaller. That is financial progress. It is a pity that it comes at such a cost.