At 2:32 pm Eastern Time, on May 6, 2010, a mutual fund complex began selling a large batch of E-mini S&P 500 futures contracts on the New York Stock Exchange. The exact number of contracts to be liquidated by this seller was based on a computer algorithm linked to the exchange’s real-time liquidity, as measured primarily by trading volumes.

The first buyers for these futures contracts were high-frequency traders – essentially computer programs that scooped them up and more or less instantaneously re-sold them for a few pennies’ profit to other traders. But because stocks were already down by about 4% that day, based on worries about the European debt crisis, there weren’t many fundamental buyers in the market, so most of the onward buyers were also computer-controlled high-frequency traders, which passed them on to other high-frequency traders in no more than a second or two, each time driving the price down very slightly, while trading volume rapidly climbed.

The seller’s algorithm, however, interpreted the increase in trading volume as a sign of market liquidity, so the more high-frequency trades occurred, the more futures contracts the seller offered for sale, generating ever higher volumes in a vicious cycle in which stock prices for a wide variety of companies careened spectacularly out of control. Within a mere 13 minutes the Dow Jones had suffered its worst intra-day decline in history (nearly a thousand points), and a trillion dollars of value had been wiped out! A number of trades of individual shares had taken place at clearly ridiculous values – as low as a penny a share, and as high as $100,000 a share.

At 2:45 pm, however, an automatic safety measure at the exchange kicked in, and trading on the E-mini was halted for five seconds. This interruption, as brief as it was, broke the cycle, and when trading resumed prices everywhere began a return to normalcy, rapidly returning the markets to close to their former levels. Dubbed the “Flash Crash,” this market turmoil in the real world was caused by computers, trading on their own at lightning speed.

If you want to get a feeling for how fast a true, computer-based artificial intelligence (AI) might get out of hand and wreak terrible destruction, by accident, then studying how the Flash Crash happened in just a few minutes isn’t a bad place to start. Oxford philosophy professor Nick Bostrom, in his new book Superintelligence: Paths, Dangers, Strategies, argues that even though it didn’t truly stem from any type of AI, because the programs doing the trading were based on simple, human-designed algorithms and not particularly smart, we can still draw some lessons.

First, “interactions between individually simple components…can produce complicated and unexpected effects.” Computers trading among themselves represent a complex system with feedback loops and recursive, unpredictable outcomes.

Second, even though “smart professionals might give an instruction to a program based on a sensible-seeming and normally sound assumption (e.g. that trading volume is a good measure of market liquidity),” we have to remember that the program itself has no common sense and will continue operating on that assumption even in unforeseen circumstances when the assumption turns out to be invalid or irrelevant.

And finally, Bostrom suggests, even though automation contributed to the Flash Crash, “it also contributed to its resolution. The pre-preprogrammed stop order logic, which suspended trading when prices moved too far out of whack, was set to execute automatically because it had been correctly anticipated that the triggering events could happen on a timescale too swift for humans to respond.” In other words, it’s important to build these kinds of safety triggers into future computer software, because when things go wrong, there won’t be time for human common sense to prevail.

One additional lesson can be drawn from the Flash Crash, as well – a lesson that only became evident after Bostrom’s book had already gone to press. Earlier this year, five years after the Flash Crash, blame has finally been leveled at a single individual: a British subject, one Navinder Singh Sarao, who apparently triggered the whole market meltdown with his own trading patterns. According to Bloomberg, Sarao modified some commercially available trading software in order to be able to “rapidly place and cancel orders automatically” and then he placed orders for roughly $200 million of E-mini futures contracts, replacing or modifying these orders some 19,000 times before they were all canceled. The U.S. is now seeking to extradite Sarao on 22 criminal counts alleging market manipulation and fraud.

But who exactly is Navinder Sarao? What sort of financial titan could single-handedly push the world securities market down by a trillion dollars in value, even for a few minutes? Sarao is a day trader in business on his own. He lives with his parents in a modest stucco bungalow in a Sikh community on the west side of London. He doesn’t even know how to drive.

So the final lesson we should take from this episode is that when and if true artificial intelligence actually does become possible, we not only need to take measures ensuring that it doesn’t manipulate or harm humans, but we also need to make sure that humans aren't allowed to manipulate it.