For more than fifteen years, I spent a considerable amount of my professional time in the company of oil men. For years I ate lunch with them, traveled with them to places like Scotland and Russia, listened to their war stories over drinks, watched them unfurl seismic charts on board tables, read their budgets, and marveled at their forecasts — all of which predicted finding the next Prudhoe Bay, North Sea, Bass Strait, or Caspian Sea, no matter where they looked. In one meeting, I heard of a vast store of gas under, alas, the walls of old Jerusalem.

The oil men of my acquaintance had names like Swede, Red, Junior, Clint, and Roge. To a man they believed that oil could be found through satellite images, wildcat drilling, 3-D seismic surveys, or, in one case, a careful reading of select Biblical passages.

All they ever needed to realize their dreams was about $25 million, a few drilling rigs, a handful of outside experts (all of whom charged $2000 a day to pour cement, decode seismic, or spud wells), and then to be left alone until the crude came bubbling to the surface, as it does in the opening scenes of "The Beverly Hillbillies".

The stories of British Petroleum's Deepwater Horizon have reminded me that the industry runs on the fumes of dreams, what Sigmund Freud called “the future of an illusion.”

According to the mainstream media accounts, which read like a Frank Norris novel, the petroleum industry is an octopus that has wrapped its stranglehold around all aspects of America’s economic and political life. It’s methodical, logical, all-encompassing, and evil.

In practice, most oil companies are lotteries with gas stations.

Yes, the Gulf oil spill is a cautionary tale of what happens when wily corporations strong-arm regulators, scoff at environmental impact statements, and ignore safety regulations to bleed the world of its precious energy reserves, all for extortionate profits.

That said, the Deepwater Horizon sounds like every oil deal I ever heard described, with the exception that the shareholders and management of BP were willing to fund the diving rods with millions in actual front money. In most cases, petroleum deals do not get past the sketches drawn on restaurant napkins.

The only way to go into the oil business, at least on the upstream (exploration) side, is to think big. Downstream (refining, gas stations, Jiffy Lube and the like) is closer to banking or life insurance, a narrow-margin turnover business. Upstream is the stuff of wildcat dreams, where the right drilling rig (which can cost $100,000 a day) and the right well (some as deep as several miles) will unlock a gusher.

Large oil companies, however, rarely go it alone when developing a new field. In places like the North Sea and the Caspian, they syndicate participations to other oil companies in order to reduce capital requirements and spread around the risk.

With the Deepwater Horizon, apparently BP thought they had a sure thing and held on to a majority. If you're tapping into Paradise, why bring in outside angels?

Doubt is not an emotion that I associate with oil men; think of BP’s chief executive, Tony Hayward, who said, “I think the environmental impact of this disaster is likely to have been very, very modest,” and then rode out the media storms on his yacht. I am sure BP’s executives who planned the Gulf drilling are no different from those that I met who preached the gospel of untapped petroleum wealth in Sudan, Mongolia, the Democratic Republic of Congo, and Albania.

Whenever I would ask about the risks or costs of a venture, the oil men at the table would look at me as if I were shoe salesman who was trying to understand the finer points of neuroscience. My job as a banker was to find the money to sink the drills. Beyond that, I was on a “need not to know” basis.

How often do oil men hit oil? Obviously, many wells hit pay dirt. Nonetheless, I spent fifteen years following drilling adventures and no one I came across ever shouted “Eureka!”. Instead, I was introduced to the concept of the “dry hole,” a noble but unsuccessful effort to tap into the riches of the earthen core, which costs about eight million dollars a whack.

When a hole comes back empty, oil men generally agree that failure is key to the industry, that the faults were not their own, and that next time, with another eight or eighty million to sink into the ground, they will do things differently, such as actually study the geological formations, consult earlier drillers, prepare a budget, or call some top notch oilman who has retired to Wyoming. Many oil meetings end with someone saying, “Let’s get Swede over here.”

As best I could tell, drilling is done on a wing and a prayer. I am sure this was the case for BP as much as it was for my biblical geologists, despite all of the “beyond petroleum” spin, Maybe the company did not consult the Bible after it anchored the Deepwater Horizon, but otherwise it might well have followed the script from the Book of Revelation (‘The sea is turning into blood and everything is dying’).

Another central belief of the petroleum industry is that losses measure an oil man as much as does success. Bankers hate bad loans, and retailers loathe getting undercut. Oil men, however, take a fair amount of pleasure in losing huge sums of money. It’s part of the culture to have lost $100 million in the Caspian or $300 million in the Black Sea. It shows the world that their pockets are as deep as their drill bits.

I am not saying that BP wanted its well to blow out or its crude to pollute the Gulf of Mexico, although failure does have its rewards. I am sure that many oil men, including a number inside BP, are having what might be called “a good spill.” It has been as munificent as holy water to all the consultants earning $4000 a day for their opinions on caps, relief wells and blowout preventers.

If properly maintained, wells produce oil and gas for years. That does not take away the go-for-broke, extreme risk element that is critical to the industry’s emotional DNA, and probably it’s success.

I remember one wildcat deal in the former Soviet republic of Georgia. In meeting after meeting, specialists from Texas, Aberdeen or Egypt poured over budget projections showing billion dollar profits, the largest gas field in Central Asia, the next East Texas in Tbilisi.

It was a liquid gold mine, and for “$2 million, $5 million, okay, $15 million” it could be reached with a few rigs shipped in from Israel. (As John the Geologist wrote in Revelations: “I will give unto him that is athirst of the fountain of the water of life freely.”)

In the end, the investors put up $45 million for three dry holes. The reasons for failure were endless: the seismic was old, the Soviets had ruined the wells, the reservoirs were fractured, water had diluted the oil, the Georgians were siphoning the oil into their cars, you name it. After that, the drillers blamed the owners for “failing to support the project” and walked away.

Who was right? I have no idea, although I was impressed by the insight of one executive who had gone to many planning meetings. Earlier in his career, he might even have worked for BP. “Well,” he said, “they didn’t find any oil or gas, but they certainly got their $40 million worth of fun.”

U.S. Coast Guard photo by Petty Officer 3rd Class Patrick Kelley of Deepwater Horizon Flaring Operation, posted by DVIDSHUB

Matthew Stevenson is the author of Remembering the Twentieth Century Limited, winner of Foreword’s bronze award for best travel essays at this year's BEA. He is also editor of Rules of the Game: The Best Sports Writing from Harper's Magazine . He lives in Switzerland.