AFTER oil hit its recent record of $135 a barrel, consumers and politicians started to lash out in every direction. Fishermen in France have been blockading ports and pouring oil on the roads in protest. British lorry drivers have paraded coffins through London as a token of the imminent demise of the haulage industry. In response, Gordon Brown, Britain's prime minister, is badgering oil bosses to increase production from the North Sea, while Nicolas Sarkozy, the president of France, wants the European Union to suspend taxes on fuel.

In America, too, politicians are haranguing oil bosses and calling for tax cuts. Congress has approved a bill to prevent the government from adding to America's strategic stocks of oil, and is contemplating another to enable American prosecutors to sue the governments of the Organisation of the Petroleum Exporting Countries (OPEC) for market manipulation.

But the most popular scapegoats are “speculators” of the more traditional sort. OPEC itself routinely blames them for high prices. The government of India is so sure that speculation makes commodities dearer that it has banned the trading of futures contracts for some of them (although not oil). Germany's Social Democratic Party proposes an international ban on borrowing to buy oil futures, on the same grounds. Joe Lieberman, chairman of the Senate's Homeland Security Committee, is also mulling regulation of some sort, having concluded that “speculators are responsible for a big part of the commodity price increases”. The assumption underlying such ideas is that a bubble is forming, and that if it were popped, the price of oil would be much lower.

Others assume the reverse: that the price is bound to keep rising indefinitely, since supplies of oil are running short. The majority of the world's crude, according to believers in “peak oil”, has been discovered and is already being exploited. At any rate, the size of new fields is diminishing. So production will soon reach a pinnacle, if it has not done so already, and then quickly decline, no matter what governments do.

As different as these theories are, they share a conviction that something has gone badly wrong with the market for oil. High prices are seen as proof of some sort of breakdown. Yet the evidence suggests that, to the contrary, the rising price is beginning to curb demand and increase supply, just as the textbooks say it should.

Stocks, bonds and barrels

Those who see speculators as the culprits point to the emergence of oil and other commodities as a popular asset class, alongside stocks, bonds and property. Ever more investors are piling into the oil markets, the argument runs, pushing up the price as they do so. The number of transactions involving oil futures on the New York Mercantile Exchange (NYMEX), the biggest market for oil, has almost tripled since 2004. That neatly mirrors a tripling of the price of oil over the same period.

But Jeffrey Harris, the chief economist of the Commodity Futures Trading Commission (CFTC), which regulates NYMEX and other American commodities exchanges, does not see any evidence that the growth of speculation in oil has caused the price to rise. Rising prices, after all, might have been stimulating the growing investment, rather than the other way around. There is no clear correlation between increased speculation and higher prices in commodities markets in general. Despite a continuing flow of investment in nickel, for example, its price has fallen by half over the past year.

By the same token, the prices of several commodities that are not traded on any exchange, and are therefore much harder for speculators to invest in, have risen even faster than that of oil. Deutsche Bank calculates that cadmium, a rare metal, has appreciated twice as much as oil since 2001, for example, and the price of rice has risen fractionally more.

Investment can flood into the oil market without driving up prices because speculators are not buying any actual crude. Instead, they buy contracts for future delivery. When those contracts mature, they either settle them with a cash payment or sell them on to genuine consumers. Either way, no oil is hoarded or somehow kept off the market. The contracts are really a bet about which way the price will go and the number of bets does not affect the amount of oil available. As Mr Harris puts it, there is no limit to the number of “paper barrels” that can be bought and sold.

That makes it harder for a bubble to develop in oil than in the shares of internet firms, say, or in housing, where the supply of the asset is finite. Ultimately, says David Kirsch of PFC Energy, a consultancy, there is only one type of customer for crude: refineries. If speculators on the futures markets get carried away, pushing prices so high that refineries run at a loss, they will simply shut down, causing the price to fall again. Moreover, speculators do not always assume that prices will rise. As recently as last year, the speculative bears on NYMEX outweighed the bulls.

There is, admittedly, a growing category of inherently bullish investment funds that seek to track commodity-price indices, in which oil is usually the biggest component. Politicians have begun to denounce these “index funds”, since they make money for their investors only if prices rise. According to Mr Lieberman, they have grown in value from $13 billion to $260 billion over the past five years. This surge of investors betting on rising prices, many observers contend, has become a self-fulfilling prophecy, helping to push prices ever higher and thus attract yet more investment.

But Bob Greer, of PIMCO, an asset-management firm, argues that even index funds make unlikely suspects. For one thing, they too invest in futures, rather than in physical supplies of oil. So every month, they must trade contracts that are about to fall due for ones that will not mature for several months. That makes them big sellers of oil for prompt delivery.

What is more, their growth is not as impressive as it first appears. Paul Horsnell of Barclays Capital, an investment bank, puts the total value of index funds and other similar investments at $225 billion. That is less than half the market capitalisation of Exxon Mobil, he points out, and a tiny fraction of the $50 trillion-odd of transactions in the oil markets each year. Although index funds have grown quickly, that growth stems in large part from the rise in value of the futures they hold, rather than from fresh investment flows. He estimates that index funds swelled by $13 billion in the first quarter of this year, for example, of which all but $2 billion derives from the rise in commodity prices.

Back to basics

Mr Harris of the CFTC, for one, believes that the oil price is still a function of supply and demand. For the past few years, the world's production capacity has grown only sluggishly. Meanwhile, demand, especially from the developing world, has been growing faster. So there is hardly any slack in the system. Only Saudi Arabia and the United Arab Emirates are thought to be able to increase their output from today's levels, and even then, there are doubts, since Saudi Arabia, in particular, is secretive about the state of its oil industry.

That leaves the oil market at the mercy of even small disruptions to supply. Prices tend to jump each time militants sabotage an oil pipeline in Nigeria, bad weather threatens production in the Gulf of Mexico, or political clouds gather over the Persian Gulf.

The problem is exacerbated by a growing mismatch between the type of oil being produced and the refineries that must process it. The most common benchmark prices, including the one used in this article, refer to “light” crude, the least viscous sort, which produces the most petrol and diesel when refined. “Heavy” oil, by contrast, yields more fuel oil, which is used mainly for heating.

At the moment, diesel is in short supply and there is a glut of fuel oil. That makes processing heavy oil unprofitable for some refineries, since the gains from diesel are outweighed by losses on fuel oil. As refineries turn instead to lighter grades, it pushes their prices yet higher. The discount on heavier crudes has risen to record levels. But even then, points out Ed Morse, of Lehman Brothers, another investment bank, Iran is having trouble selling the stuff. It is storing huge quantities of unsold oil on tankers moored off its coast.

Presumably, Iran and other heavy-oil producers will eventually be obliged to drop prices far enough to make processing the stuff worth refiners' while. In the longer run, more refineries will invest in the equipment needed to crack more diesel out of heavy oil. Both steps will, in effect, increase the world's oil supply, and so help to ease prices.

But improving an existing refinery or building a new one is a slow and capital-intensive business. Firms tend to be very conservative in their investments, since refineries have decades-long life-spans, during which prices and profits can fluctuate wildly. It can also be difficult to find a site and obtain the right permits—one of the reasons why no new refineries have been built in America for over 30 years. Worse, new kit is becoming ever more expensive. Cambridge Energy Research Associates (CERA), a consultancy, calculates that capital costs for refineries and petrochemical plants have risen by 76% since 2000.

Much the same applies to the development of new oilfields. CERA reckons that the cost of developing them has risen even faster—by 110%. At the same time, oilmen remain scarred by the rapid expansion of output in the late 1970s, in response to previous spikes in prices, that led to a glut and so to a prolonged slump. Exxon Mobil claims that it still assesses the profitability of potential investments using the same assumptions about the long-term oil price as it did at the beginning of the decade, for fear that prices might tumble again. Environmental concerns are also an obstacle: America, for one, has banned oil production off most of its coastline.

Increasing nationalism on the part of oil-rich countries is adding to the difficulties. Geologists are convinced that there is still a lot of oil to be discovered in the Middle East and the former Soviet Union, but governments in both regions are reluctant to give outsiders access. Elsewhere, the most promising areas for exploration are also the most technically challenging: in deep water, or in the Arctic, or both. Although there have been big recent discoveries in such places, they will take longer to develop, and costs will be higher. The most expensive projects of all involve the extraction of oil from bitumen, shale and even coal, through elaborate processing. The potential for these is more or less unlimited, although analysts put the costs as high as $70 a barrel—more than the oil price this time last year.

Nonetheless, PFC Energy has examined projects that are already under way, and concluded that global oil production will grow by over 3m barrels a day (b/d) over the course of this year and next. In particular, it expects production outside OPEC to grow by about 500,000 b/d both years—a marked increase from the near stagnation of recent years.

Meanwhile, the high price is clearly beginning to crimp demand. The growth in global consumption last year was barely a quarter what it was in 2004 (see chart); this year, it is likely be even lower. In rich countries (or at least among the members of the Organisation for Economic Co-operation and Development (OECD), a rough proxy), the effect is even more pronounced. Consumption has been falling for the past two and a half years.

Poorer countries' demand for oil is still rising, albeit at a slowing pace. That is partly because their economies are growing faster, and partly because their consumers are shielded from the rising price through subsidies. But the increasing expense of such measures is forcing governments to water them down or scrap them altogether (see article). That, in turn, should further sap consumption.

Oil pique

China's growing thirst for oil is often put forward as one of the main factors behind today's higher oil prices. Demand for diesel there, for example, rose by over 9% in the year to April. But Mr Morse argues that such growth might not last. The government has ordered oil firms to increase their stocks of fuel by 50% to be sure there are no embarrassing shortages during the Olympics. It is also planning to run some power plants near Beijing on diesel rather than coal, in an attempt to reduce pollution during the games. These measures are helping to boost China's demand for diesel, but the effect will be transitory.

In the short run, neither demand for nor supply of oil is very elastic. It takes time for people to replace their old guzzlers with more fuel-efficient cars, or to switch to jobs with shorter commutes, or to move closer to public transport. By the same token, it can take ten years or more to develop an oilfield after its discovery—and that does not include the time firms need to bolster their exploration units.

Gary Becker, an economist at the University of Chicago, has calculated that in the past, over periods of less than five years, oil consumption in the OECD dropped by only 2-9% when the price doubled. Likewise, oil production in countries outside OPEC grew by only 4% every time the price doubled. But over longer periods, consumption dropped by 60% and supply rose by 35%. The precise numbers may be slightly different this time round, but the pattern will be the same.