Between 2006 and 2015, the energy world was turned upside-down by an epic development in the oil industry few had foreseen. From the low point, in 2006, when it imported 60% of its oil, the US became an oil powerhouse – eclipsing both Saudi Arabia and Russia – and by the end of 2015, was the world’s largest producer of natural gas.

This remarkable transformation was brought about by American entrepreneurs who figured out how to literally force open rocks often more than a mile below the surface of the earth, to produce gas, and then oil. Those rocks – called shale, source rock or tight rock, and once thought to be impermeable – were opened by combining two technologies: horizontal drilling, in which the drill bit can travel more than two miles horizontally, and hydraulic fracturing, in which fluid is pumped into the earth at a high enough pressure to crack open hydrocarbon-bearing rocks, while a so-called proppant, usually sand, holds the rocks open a sliver of an inch so the hydrocarbons can flow. A fracking entrepreneur likens the process to creating hallways in an office building that has none – and then calling a fire drill.

In November 2017, US production topped the 10m barrel-a-day record set in 1970, back in the last gasp of the legendary oil boom. This year, it is expected to reach almost 11m barrels a day, according to the US Energy Information Administration. The Marcellus Shale, which stretches through northern Appalachia, could be the second-largest natural gas field in the world, according to geologists at Penn State. Shale gas now accounts for more than half of total US production, according to the EIA, up from almost nothing a decade ago.

The apparent new era of American energy abundance has already had a profound impact around the world. Economies that were dependent on the high price of oil, from Russia to Saudi Arabia, have begun to struggle. The situation would have been unthinkable in the pre-2014 world of $100-a-barrel oil, and is playing out in strange and unpredictable ways.

Since the 1970s, US presidents from Gerald Ford to both Bushes emphasised the importance of “energy independence”, although the country had in fact become more and more dependent, particularly on the Middle East. Under the Trump administration, the longstanding dream of America’s energy independence has taken a grander, more muscular turn. Secretary of the interior Ryan Zinke talks about opening more federal lands, including national parks, to drilling in order to ensure “energy dominance”.

“We’ve got underneath us more oil than anybody, and nobody knew it until five years ago,” Trump told the press aboard Air Force One in the summer of 2017. “And I want to use it. And I don’t want that taken away by the Paris accord. I don’t want them to say all of that wealth that the United States has under its feet, but that China doesn’t have and that other countries don’t have, we can’t use.”

But the shale success story nearly became a disaster. While to date, most of the complaints about fracking have focused on environmental concerns, there’s a bigger and far less well known reason to doubt the most breathless predictions about America’s future as an oil and gas giant. The fracking of oil, in particular, rests on a financial foundation that is far less secure than most people realise.

Because so few fracking companies actually make money, the most vital ingredient in fracking isn’t chemicals, but capital, with companies relying on Wall Street’s willingness to fund them. If it weren’t for historically low interest rates, it’s not clear there would even have been a fracking boom at all.

‘You can make an argument that the Federal Reserve is entirely responsible for the fracking boom,” one private-equity titan told me. That view is echoed by Amir Azar, a fellow at Columbia University’s Center on Global Energy Policy. “The real catalyst of the shale revolution was the 2008 financial crisis and the era of unprecedentedly low interest rates it ushered in,” he wrote in a recent report. Another investor put it this way: “If companies were forced to live within the cash flow they produce, US oil would not be a factor in the rest of the world, and would have grown at a quarter to half the rate that it has.”

Worries about the financial fragility of the fracking revolution have simmered for some time. John Hempton, who runs the Australia-based hedge fund Bronte Capital, recalls having debates with his partner as the boom was just getting going. “The oil and gas are real,” his partner would say. “Yes,” Hempton would respond, “but the economics don’t work.”

Thus far, the fracking industry has been more resilient than anyone would have dreamed. But questions about the sustainability of the boom are no longer limited to a small set of skeptics. Those doubts now extend to the boardrooms of some big investors, as well as to the executive suites of at least a few of the fracking companies themselves. The fracking boom has been fuelled mostly by overheated investment capital, not by cash flow.

If the story of the fracking boom has a central character, it’s Aubrey McClendon, the founder of Chesapeake Energy, a startup that grew into a colossus. For a brief moment in history, he most represented US fracking to the world. No one was more right and more wrong, no one bolder in his predictions or more spectacular in his failures, no one more willing to risk other people’s money and his own, than McClendon; or, as one banker who knew McClendon well put it: “The world moves when people who like risk take action.”

“He was the good face of the industry – the passion, the creativity, the daring,” another former investment banker told me. “But he was also the bad face.” And that duality makes him a perfect personification of the US fracking revolution.

Fracking tycoon Aubrey McClendon, who was killed in a car crash in 2016. Photograph: Layne Murdoch/NBAE/Getty Images

McClendon’s death, like his legacy, was hotly contested. On 2 March 2016, just after 9am, McClendon slammed his Chevrolet Tahoe SUV into a concrete viaduct under a bridge on Midwest Boulevard in Oklahoma City, and died instantly. He was speeding, wasn’t wearing a seatbelt, and didn’t appear to make any effort to avoid the collision. Just one day earlier, a federal grand jury had indicted him for violating antitrust laws during his time as the CEO of Chesapeake Energy. Investigators ultimately ruled his death an accident, but rumours of suicide persist to this day. As Capt Paco Balderrama of the Oklahoma City police told the press: “We may never know 100% what happened.”

In the fall of 2008, Forbes had ranked McClendon No 134 on its list of the 400 richest Americans, with an estimated net worth of more than $3bn. But because he borrowed so much money and secured business loans with personal guarantees, lawyers were still wrangling over the remains of his estate two years after his death, trying to figure out which debts would be paid – from the $500,000 he owed the Boy Scouts of America to the $465m he owed a group of Wall Street creditors, including Goldman Sachs. Wall Street’s vultures – the hedge funds that invest in distressed debt – had descended, buying the debt for less than 50 cents on the dollar, essentially rendering a judgment that the claims wouldn’t be paid in full. If McClendon did die broke, it wouldn’t have been out of character. During his years as an oil and gas tycoon, he fed on risk, and was as fearless as he was reckless. He built an empire that at one point produced more gas than any American company except ExxonMobil. Once, when an investor asked on a conference call, “When is enough?”, McClendon answered bluntly: “I can’t get enough.”

Many think that without McClendon’s salesmanship and his astonishing ability to woo investors, the world would be a far different place today. Stories abound about how, at industry conferences, executives from oil majors like Exxon would find themselves speaking to mostly empty seats, while people literally fought for space in the room where McClendon was holding forth. “In retrospect, it was kind of like Camelot,” said Henry Hood, Chesapeake’s former general counsel, who worked at the company, initially as a consultant, from 1993 until the spring of 2013. “There was a period of time that will never be duplicated, with a company that will never be duplicated.”

“America’s Most Reckless Billionaire,” Forbes once called McClendon, and for many in the industry, that headline defined the man. But if it was a con, he was conning himself, too. Because he believed. He was, in many ways, the embodiment of a transformation that has changed the face of not just the oil and gas industries, but of geopolitics as well.

In the darkest days of the collapse of oil prices in the mid-1980s, McClendon, as ever undeterred, saw an opportunity in assembling packages of drilling rights – for gas, not oil – either to be sold to bigger companies or to be drilled. In the mere existence of that opportunity, America is almost unique, because it is one of the few countries where private citizens, rather than governments, own the mineral rights under their properties. In order to drill, you just have to persuade someone to give you a lease. McClendon became what’s known in the oil and gas business as a “land man” – the person who negotiates the leases that allow for drilling. That, it turned out, would make him the perfect person for the new world of fracking, which is not so much about finding the single gusher as it is about assembling the rights to drill multiple wells. “Landmen were always the stepchild of the industry,” he later told Rolling Stone. “Geologists and engineers were the important guys – but it dawned on me pretty early that all their fancy ideas aren’t worth very much if we don’t have a lease. If you’ve got the lease and I don’t, you win.”

In 1983, when McClendon was just 24 years old, he went into partnership with another Oklahoman named Tom Ward, “doing deals for scraps of land in Oklahoma, faxing each other in the middle of the night,” Ward said to Rolling Stone. Six years later, the two formed Chesapeake Energy, which was named after the beloved bay where McClendon’s family vacationed. They seeded it with a $50,000 investment.

Neither Ward nor McClendon were technological pioneers. That distinction, most people agree, goes to a man named George Mitchell, who drew on research done by the government to experiment on the Barnett Shale, an area of tight rock in the Fort Worth basin of North Texas. Using a combination of horizontal drilling and hydraulic fracturing, Mitchell’s team cracked the code for getting gas out of rock that was thought to be impermeable.

“As oxygen is to life, capital is to the oil and gas business,” said Andrew Wilmot, a Dallas-based mergers and acquisitions adviser to the oil and gas industry at Purposed Ventures. “This industry needs capital to fire on all cylinders, and the founder and father of raising capital for shale in the US is Aubrey McClendon.”

“To be able to borrow money for 10 years and ride out boom-and-bust cycles was almost as important an insight as horizontal drilling,” McClendon, with typical immodesty, said to Rolling Stone.

A fracking site in Texas in 2017. Photograph: Bloomberg via Getty

On 12 February 1993 – a day McClendon would later describe as the best of his career – he and Ward took Chesapeake public. They did so despite the fact that their accounting firm, Arthur Andersen, had issued a “going concern” warning, meaning its bean-counters worried that Chesapeake might go out of business. So McClendon and Ward simply switched accounting firms. “Tom and I were 33-year-old landmen at the time, and most people didn’t think we had a clue what we were doing, and probably in hindsight they were at least partially right,” McClendon told an interviewer in 2006.

In the decade before 2004, Chesapeake spent around $6bn acquiring properties, companies and leases. McClendon, who would later call these years the “the great North American land grab”, developed a reputation among his peers for overpaying. His aggressiveness didn’t endear him to the old-time oil men. “Everyone in Midland hated Chesapeake,” one said. “They came out here when land was leasing for $200-$300 an acre. All of a sudden, Chesapeake was paying $2,000-$3,000. They got in some good places because they shut everyone else out. Their attitude was: ‘We are Chesapeake, get out of our way.’”

“[McClendon’s] aggressive style ruffled some feathers in the industry,” Andrew Wilmot said. “He went guns blazing, and drove up the prices. That made some people millionaires, but it wreaked havoc on others.”

McClendon went on a corporate spending spree that would have put today’s Silicon Valley chieftains to shame. “Asking me what to do with extra cash is like asking a fraternity boy what to do with the beer,” McClendon told Natural Gas Intelligence in 2005. Nor was he frugal when it came to his personal life. He acquired multimillion-dollar mansions and resorts in Oklahoma, Bermuda, Maui, Vail, on Lake Michigan, and even in Minnesota. He had one of the best wine collections in the world.

To Wall Street investors, McClendon was delivering on what they wanted most: consistency and growth. His pitch was that fracking had transformed the production of gas from a hit-or-miss proposition to one that operated with an on and off switch. It was manufacturing, not wildcatting. He became a flag-waver for natural gas – “Mr Gas”, as Fortune magazine once called him.

“Aubrey was the first one to say, ‘Let’s create demand,’” Chesapeake’s Henry Hood said.

Back in 2003, when McClendon was just getting started, the consensus view had been that the US was running out of natural gas. It became a fixation for Alan Greenspan, the once-revered chair of the Federal Reserve, who warned Congress during a rare appearance that the shortage and rising cost of gas could hurt the American economy. Greenspan recommended that the US build terminals to accept deliveries of liquefied natural gas from other countries. “We see a storm brewing on the horizon,” said Billy Tauzin, a Republican representative from Louisiana and the then-chairman of the Energy and Commerce Committee. Such fears eventually helped push through the Energy Policy Act of 2005, which exempted natural gas drillers from having to disclose the chemicals used in hydraulic fracturing, thus averting costly regulatory oversight.

As fracking took off, McClendon began telling anyone who would listen that the US had enough natural gas to last more than 100 years. He quietly financed a campaign called “Coal is Filthy”, and he argued that converting 10% of US vehicles to run on natural gas in the next 10 years would be the fastest, cheapest way to free the country from dependency on foreign oil. He was adamant that employees should drive cars fuelled by compressed natural gas. For a man steeped in the industry’s history of boom and bust, McClendon had by now convinced himself that natural gas prices would never fall. In August 2008, he predicted that prices would stay in the $8-$9 range for the foreseeable future. “He had a very, very strong point of view about gas,” said one banker who knew him since the early 1990s. “By the way, he was basically wrong for the last 30 years.”

McClendon’s bullish view on prices became the conventional wisdom in energy markets. In 2007, the supposedly smartest investors in the world – among them Goldman Sachs and the takeover titan KKR – structured their massive $45bn buyout of a utility called TXU in a way that was essentially a bet that natural gas prices, then around $7, were set to rise significantly.

At the same time, Vladimir Putin was making similar bets. In an attempt to set up a cartel for gas, the Russian premier hosted a group of gas-producing countries, including Algeria, Iran, and Venezuela, in Moscow. The US was not among them. “Costs of exploration, gas production and transportation are going up,” Putin said. “It means the industry’s development costs will skyrocket. The time of cheap energy resources, cheap gas, is surely coming to an end.”

When the going got rough, McClendon had always survived by borrowing yet more money to acquire more properties. “Simply put, low prices cure low prices as consumers are motivated to consume more and producers are compelled to produce less,” he wrote in Chesapeake’s 1998 annual report. But he had forgotten the flipside of that industry truism. Time and again, in commodity markets, high prices encourage more producers to produce, creating a surplus, that then crushes prices – and producers. “He was right that shale changed the world,” said one longtime gas man. “He should have listened to himself.”

The price of natural gas began to plunge in 2012, and in 2014, the price of oil followed suit. Falling prices quickly exposed the weak underbelly of US shale – its high costs and ravenous need for capital. Once-booming US production hit the skids. The so-called rig count – the number of rigs drilling for oil and gas at a given time – fell from 1,920 rigs in late 2014 to a low of 480 in early 2016. “We think it likely that to find a lower level of activity would require going back to the 1860s, the early part of the Pennsylvania oil boom,” Paul Hornsell, head of commodities research for Standard Chartered bank, wrote in a research note. By mid-2016, US oil production had declined by 1m barrels a day.

One after another, debt-laden companies began to declare bankruptcy, with some 200 of them eventually going bust. In a report released in the fall of 2016, credit rating agency Moody’s called the corporate casualties “catastrophic”. “When all the data is in, including 2016 bankruptcies, it may very well turn out that this oil and gas industry crisis has created a segment-wide bust of historic proportions,” said David Keisman, a Moody’s senior vice-president.

Some of those who had bought assets from McClendon and others in the heyday also began to write down the value of what they had purchased. Statoil, the Norwegian energy giant, wrote down the value of its shale and Canadian oil sands assets by $4bn; Royal Dutch Shell reported a write-down of more than $8bn. Most prominent was Australia’s BHP Billiton, which had spent $5bn investing with Chesapeake in the Fayetteville shale and ploughed another $15bn into the purchase of Houston-based Petrohawk. BHP put all the assets on the block in the fall of 2014, but found no buyers, and eventually wrote off more than $7bn – which begat the phrase “pulling a BHP”.

As one investor put it: “All of the acquisitions of shale assets done by the majors and by international companies have been disasters. The wildcatters made a lot of money, but the companies haven’t.”

As shale companies slashed their budgets, fracking equipment was idled – research firm IHS Markit reported in 2016 that close to 60% of the fracking equipment in the US was inactive. Shale companies and oilfield service companies laid off workers. All told, the global oil and gas industry shed almost half a million jobs during the bust, according to consulting firm Graves & Co.

The shale boom towns suddenly resembled their California counterparts after the gold rush. In the Cline shale east of Midland in Texas, Devon Energy decreased its rig activity and let its leases expire, citing “a lot of variability” in the formation. In the town of Sweetwater, “ambitions are fading fast as the plummeting price of oil causes investors to pull back, cutting off the projects that were supposed to pay for a bright new future,” wrote the Associated Press in early 2015. “Now the town of 11,000 awaits layoffs and budget cuts and defers its dreams.”

By nearly all accounts, the shale boom had gone bust. In early 2016, non-investment grade energy bonds – the shale industry’s rocket fuel – yielded 25%, five times what they had a year and a half earlier, indicating a wildly elevated level of risk. “This has the makings of a gigantic funding crisis” for energy companies, William Snyder, the head of Deloitte’s US restructuring unit, told the Wall Street Journal in early 2016. That spring, the Kansas City Federal Reserve concluded that “current prices are too low for much long-term economic viability of shale oil production”.

Surveying the carnage in the spring of 2016, then ExxonMobil CEO Rex Tillerson told a gathering of analysts that due to the huge amount of debt most companies in the industry had accumulated, he couldn’t even find anything worth buying.

When Aubrey McClendon died in his car, colliding with a concrete wall supporting an overpass at 90mph, it was hard not to see his death as the punctuation marking the end of an era. As the Australian hedge fund manager John Hempton asked: “Is Chesapeake the model for this business? It changes the world, but it ends in tears?”

This is an edited extract from Saudi America by Bethany McLean, which will be published by Columbia Global Reports on 12 September. To buy it for £9.99, go to guardianbookshop.com or call 0330 333 6846

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.