A TECHNIQUE called “directional drilling” has transformed the energy business. Fifteen years ago the best drillers could force a well-shaft into a gentle arc. These days shafts can be drilled vertically to a depth of several kilometres—then made to turn sharply and continue horizontally for up to 12km (or 7 miles). Will Grace of Schlumberger, an oilfield services company, likens it to dropping a plumb-line from the top of the Empire State Building and then guiding it through the rear and front windscreens of every car parked in the nearby streets.

Such technology vastly increases the area one rig can cover (see diagram). For an illustration, Mr Grace points to squiggles and shadings on a computer screen in one of the 34 offices Schlumberger operates in Aberdeen, a Scottish oil city. The lines show the progress of a well completed for a Canadian oil firm a few hours earlier. It is 13,000 feet (4,000 metres) deep and has been brought to a halt 6,500 feet horizontally away from the rig, within three feet of its target. Instruments in the “drill-string”—as formerly inflexible steel drill-shafts are now called—are meanwhile transmitting dozens of additional measurements: of the radioactivity of the surrounding rock, its resistivity to electromagnetic waves, and so on. In this case, the rock gives a low radioactivity reading, which suggests that it is sand; its resistivity is high, which suggests it is oil-bearing. This is wizardry that few firms can match. And probably none is a regular oil company. Oilfield services (OFS) firms such as Schlumberger are the unsung workhorses of the oil industry. They do most of the heavy lifting involved in finding and extracting oil and gas. They are far less well-known than the oil firms that hire them, but immensely lucrative. Schlumberger, with headquarters in Paris and Houston, earned profits of $5 billion on revenues of $40 billion last year. Its market capitalisation has risen fourfold in the past decade, to $91 billion. That is bigger than several international oil companies, including ENI ($82 billion), Statoil ($75 billion) and Conoco-Philips ($71 billion). Schlumberger’s success highlights a shift in the balance of power between oil companies and their flunkeys. Until the 1990s OFS companies were far smaller and earned low margins on straightforward tasks, such as drilling vertical wells. That has changed dramatically.

With the price of oil so high, firms are scrambling to pump it out of ever more remote and costly crevices. Over the past decade the oil industry’s annual spending on exploration and production has increased fourfold in nominal terms, while oil production is up by only 12%. The big services companies, which invest heavily in technology (see chart), have been growing by around 10% a year. According to McKinsey, a consultancy, OFS companies grossed around $750 billion last year. OFS firms come in three flavours. Some make and sell expensive kit for use on drilling rigs or the seabed. These include FMC, Cameron and National Oilwell Varco, all $10-billion-plus companies. Some own and lease out drill-rigs. These companies include Transocean, Seadrill, Noble and Rowan. The third group carries out most of the tasks involved in finding and extracting oil. It is dominated by four giants: Schlumberger, Halliburton, Baker Hughes, and Weatherford International. Most of these firms were relatively small until the 1980s, when several oil companies decided that humdrum drilling chores were no longer worth doing in-house. Oil was easy then. Drilling yielded low margins that did not justify its claim on capital, so the oil majors outsourced it. This gave OFS firms space to grow. They grew even faster in the early 1990s, when a tightening oil market drove demand for new technology. This led to breakthroughs in 3D seismology and directional drilling. These breakthroughs allow oil to be sucked economically from far beneath the ocean floor, and out of depleted and formerly abandoned wells.

But such inventions do not come cheap. Schlumberger invests roughly $1 billion a year in research and development, a level it maintained even during the slump after the 2008 financial crisis. That is as much as the mighty ExxonMobil spends; as a share of sales, five times more. The big OFS companies now probably file more patent applications than the oil majors, whose technological skills are largely interpretive. (For example, an oil major may decide where and how to drill based on geophysical data provided by an OFS firm.)

The oil business is likely to grow even more dependent on brainy OFS firms. Global production from mature oilfields is falling by between 2% and 6% a year. In the North Sea it has declined by 6% a year on average since 1999. With global demand for oil growing by 1-2% a year, there are persistent fears of a supply shock. Hence the current high oil prices: even after a 20% fall in recent months, Brent Crude is now around $100 a barrel. Oil firms are searching harder in more remote places, such as the Arctic and the deep seas off Brazil. Operating in such places will require yet more snazzy technology.

With hindsight, the oil companies’ decision to outsource the grubby bits of the job looks like an opportunity squandered. It has also left the oil firms hostage to the availability of increasingly expensive and sought-after services, from advanced drilling to deepwater rigs, which a dwindling number of OFS firms can provide.

There is, at present, still a fair amount of competition in most parts of the services industry. Each big OFS firm has different strengths, and plenty of smaller ones occupy specialised niches. Yet in some areas, especially the geographically remote ones, the demand for complex services often outstrips the supply.

Even worse for the likes of Exxon and BP, this has come at a time when state-owned oil firms have been muscling onto the stage. In the past couple of decades these national oil companies have claimed the best acreage in most old oilfields. The OFS firms have helped them to do so. Where once the state-owned giants hired oil majors to do the work, now they can manage projects themselves and hire technical help directly from the services firms. This can sometimes involve a limited sharing of risk between national and OFS firms, just as in a regular joint venture between oil companies.

Schlumberger, for example, will agree to a measure of payment-for-performance in big contracts. If it can drill more oil from a well than the contract says it must, it charges a higher fee. Other services firms have gone further, taking small equity stakes in exploration projects.

Some analysts wonder how all this might hurt the oil majors. A few decades ago national oil companies had to turn to oil majors for the technology required to get the stuff out of the ground. Today, oilfield service companies offer all the necessary technology and are increasingly willing to take on some of the same risks as an oil company, notes Marcel Brinkman of McKinsey.

Still, it would be wildly premature to bid Exxon adieu. Schlumberger’s performance-based contracts are a long way short of owning reserves—something the company says it will never do. It lacks the mammoth balance-sheet that oil firms maintain to manage the huge risks in oil exploration. It also lacks Exxon’s expertise in managing huge projects. And it is reluctant to annoy its customers by competing with them. Moreover, choosing where and how to explore (another strength of the oil majors) is trickier than you might think.

Instead, Schlumberger is planning more of what it is best at: pushing the technological boundaries of extracting the black stuff. It has recently been busy making acquisitions—including of Smith International, an American drill-bit company, for $11.3 billion—which have given it know-how in most segments of exploration and production. It now hopes to re-engineer the entire process.

The prize of increased efficiencies—delivered in barrels of money, not oil—could be vast. A big deepwater drilling rig costs half a million dollars a day to rent, and can take three months to drill a complicated well. Any OFS company that can shave a few days off that time will be in the money. Drilling is thrilling, and getting more so.