The following is excerpted from Tim Harford’s new book Adapt: Why Success Always Starts With Failure.

In 1931, the British Air Ministry sent out a demanding new specification for a fighter aircraft. It was a remarkable document for two reasons. The first was that throughout its existence the Royal Air Force had been dismissive of fighters. The conventional wisdom was that bombers could not be stopped. Instead, foreshadowing the nuclear doctrine of mutually assured destruction, the correct use of air power was widely presumed to be to build the largest possible fleet of bombers and strike any enemy with overwhelming force. The second reason was that the specification’s demands seemed almost impossible to meet. Rather than rely on known technology, the bureaucrats wanted aviation engineers to abandon their orthodoxies and produce something completely new.

The immediate response was disappointing: three designs were selected for prototyping, and none of them proved to be much use. The Air Ministry briefly went so far as to consider ordering aircraft from Poland.

Even more remarkable than the initial specification was the response of the ministry to this awkward failure. One of the competing firms, Supermarine, had delivered its prototype late and well below specification. But when Supermarine approached the ministry with a radical new design, an enterprising civil servant by the name of Air Commodore Henry Cave-Browne-Cave decided to bypass the regular commissioning process and order the new plane as “a most interesting experiment.” The plane was the Supermarine Spitfire.

It’s not hard to make the case that the Spitfire was one of the most significant new technologies in history. A brilliant, manoeuvrable, and superfast fighter, the Spitfire—and its pin-up pilots, brave to the point of insouciance—became the symbol of British resistance to the bombers of the Nazi air force, the Luftwaffe. The plane, with its distinctive elliptical wings, was a miraculous piece of engineering.

“She really was a perfect flying machine,” said one pilot. A Californian who traveled to Britain to sign up for the Royal Air Force agreed: “I often marvelled at how this plane could be so easy and civilized to fly and yet how it could be such an effective fighter.”

“I have no words capable of describing the Spitfire,” testified a third pilot. “It was an aircraft quite out of this world.” (The source of these quotes is Leo McKinstry’s excellent history, Spitfire: Portrait of a Legend.)

It wasn’t just the Spitfire pilots who rated the plane. The top German ace, Adolf Galland, was asked by Hermann Göring, head of the Luftwaffe, what he required in order to break down the stubborn British resistance. “I should like an outfit of Spitfires” was the terse reply. Another German ace complained, “The bastards can make such infernally tight turns. There seems to be no way of nailing them.”

Thanks to the Spitfire, Britain’s tiny Royal Air Force defied overwhelming odds to fight off the Luftwaffe’s onslaught in the Battle of Britain. It was a dismal mismatch: Hitler had been single-mindedly building up his forces in the 1930s, while British defense spending was at historical lows. The Luftwaffe entered the Battle of Britain with 2,600 operational planes, but the RAF boasted fewer than 300 Spitfires and 500 Hurricane fighters. The wartime Prime Minister himself, Winston Churchill, predicted that the Luftwaffe’s first week of intensive bombing would kill 40,000 Londoners. But thanks in large part to the Spitfire’s speed and agility, the Germans were unable to neutralize the RAF.

This meant the Germans were unable to launch an invasion that could quickly have overwhelmed the British Isles. Such an invasion would have made D-Day impossible, denying the United States its platform to liberate France. It would likely have cost the lives of 430,000 British Jews. It might even have given Germany the lead in the race for the atomic bomb, as many of the scientists who moved to the United States to work on the Manhattan Project were living in Britain when the Spitfires turned back the Luftwaffe. Winston Churchill was right to say of the pilots who flew the Spitfires and the Hurricanes, “Never in the field of human conflict has so much been owed by so many to so few.”’

It is only a small exaggeration to say that the Spitfire was the plane that saved the free world. The prototype cost the government roughly the price of a nice house in London: 10,000 pounds.

When we invest money now in the hope of payoffs later, we think in terms of a return on our investment—a few percent in a savings account, perhaps, or a higher but riskier reward from the stock market. What was the return on Henry Cave-Browne-Cave’s investment of 10,000 pounds? Four hundred and thirty thousand people saved from the gas chambers, and denying Adolf Hitler the atomic bomb. The most calculating economist would hesitate to put a price on that.

Return on investment is simply not a useful way of thinking about new ideas and new technologies. It is impossible to estimate a percentage return on blue-sky research, and it is delusional even to try. Most new technologies fail completely. Most original ideas turn out either to be not original after all, or original for the very good reason that they are useless. And when an original idea does work, the returns can be too high to be sensibly measured.

The Spitfire is one of countless examples of these unlikely ideas, which range from the sublime (the mathematician and gambler Gerolamo Cardano first explored the idea of “imaginary numbers” in 1545; these apparently useless curiosities later turned out to be essential for developing radio, television, and computing) to the ridiculous (in 1928, Alexander Fleming didn’t keep his laboratory clean and ended up discovering the world’s first antibiotic in a contaminated Petri dish).

We might be tempted to think of such projects as lottery tickets, because they pay off rarely and spectacularly. They’re rather better than that, in fact. Lotteries are a zero-sum game—all they do is redistribute existing resources, whereas research and development can make everyone better off. And unlike lottery tickets, bold innovation projects do not have a known payoff and a fixed probability of victory. Nassim Taleb, author of The Black Swan, calls such projects “positive black swans.”

Whatever we call them, such ventures present us with a headache. They are vital, because the payoff can be so enormous. But they are also frustrating and unpredictable. Usually they do not pay off at all. We cannot ignore them, and yet we cannot seem to manage them effectively either.

It would be reassuring to think of new technology as something we can plan. And sometimes, it’s true, we can: the Manhattan Project did successfully build the atomic bomb; John F. Kennedy promised to put a man on the Moon inside a decade, and his promise was kept. But these examples are memorable in part because they are unusual. It is comforting to hear a research scientist, corporation, or government technocrat tell us that our energy problems will soon be solved by some specific new technology: a new generation of hydrogen-powered cars, maybe, or biofuels from algae, or cheap solar panels made from new plastics. But the idea that we can actually predict which technologies will flourish flies in the face of all the evidence. The truth is far messier and more difficult to manage.

That is why the story of how the Spitfire was developed against the odds offers a lesson for those of us who hope technology will solve the problems of today. It was developed in an atmosphere of almost total uncertainty about what the future of flying might be. In the previous war with Germany, which ran from 1914 to 1918, airplanes were a brand-new technology and were used mainly for scouting missions. Nobody really knew how they could most effectively be used as they matured. In the mid-1920s, it was widely believed that no airplane could exceed 260 miles per hour, but the Spitfire dived at over 450 mph. So it is hardly surprising that British air doctrine failed for such a long time to appreciate the potential importance of fighter planes. The idea of building fighters that could intercept bombers seemed a fantasy to most planners.

The Spitfire seemed especially fantastical as it fired directly forward, meaning that in order to aim at a target, the entire plane needed to change course. A design that struck many as much more plausible was a twin-seater plane with a gunner in a turret. Here are the words of one thoughtful and influential observer in 1938, one year before Germany and Britain went to war:

We should now build, as quickly and in as large numbers as we can, heavily armed aeroplanes designed with turrets for fighting on the beam and in parallel courses … the Germans know we have banked upon the forward-shooting plunging ‘Spitfire’ whose attack … if not instantly effective, exposes the pursuer to destruction.

The name of this Spitfire skeptic was the future Prime Minister, Winston Churchill. The plane he demanded was built all right, but few British schoolboys thrill to the legend of the Boulton-Paul Defiant. No wonder: the Defiant was a sitting duck.

It is easy to say with hindsight that official doctrine was completely wrong. But it would also be easy to draw the wrong lesson from that. Could ministers and air marshals really have predicted the evolution of aerial combat? Surely not. The lesson of the Spitfire is not that the Air Ministry nearly lost the war with their misconceived strategy. It is that, given that misconceptions in their strategy were all but inevitable, they somehow managed to commission the Spitfire anyway.

The lesson is variation, achieved through a pluralistic approach to encouraging new innovations. Instead of putting all their eggs in what looked like the most promising basket—the long-range bomber—the Air Ministry had enough leeway in its procedures that individuals like Air Commodore Cave-Browne-Cave could fund safe havens for “most interesting” approaches that seemed less promising, just in case—even approaches, like the Spitfire, that were often regarded with derision or despair.

In September 1835, Charles Darwin was rowed ashore from The Beagle and stepped into the breakers of the Galapagos Islands. He soon discovered some remarkable examples of how safe havens provide space for new things to develop—examples that would later lead him toward his theory of evolution through natural selection. Darwin, a meticulous observer of the natural world, noted the different species of finch that inhabited the islands. Not a single one was found anywhere outside the Galapagos archipelago, which lies in the Pacific Ocean 600 miles west of Ecuador in South America. Even more intriguingly, each island boasted a different selection of finches, all of similar size and colour but with very different beaks—some with thin, probing bills to grab insects, others with large powerful bills to crack seeds, still others adapted to eat fruit. The famous giant tortoises, too, had different species for different islands, some with a high-lipped shell to allow browsing on cactuses, those on the larger, grassier islands with a more conventional high-domed shell. This caught Darwin so unawares that he mixed up his specimens and had to ask the island’s vice-governor to unscramble them; Galapagos tortoises are like no other tortoise on earth, so it took Darwin a long time to figure out that there were several distinct species. When Darwin turned his attention to Galapagan plants, he discovered the same story yet again. Each island had its own ecosystem.

The Galapagos Islands were the birthplace of so many species because they were so isolated from the mainland and, to a lesser degree, from each other. “Speciation”—the divergence of one species into two separate populations—rarely happens without some form of physical isolation, otherwise the two diverging species will interbreed at an early stage, and converge again.

Innovations, too, often need a kind of isolation to realise their potential. It’s not that isolation is conducive to having ideas in the first place: Gene mutations are no more likely to happen in the Galapagos than anywhere else, and as many people have observed, bright ideas emerge from the swirling mix of other ideas, not from isolated minds. Jane Jacobs, the great observer of urban life, looked for innovation in cities, not on Pacific islands. But once a new idea has appeared, it needs the breathing space to mature and develop so that it is not absorbed and crushed by the conventional wisdom.

This idea of allowing several ideas to develop in parallel runs counter to our instincts: We naturally tend to ask, “What is the best option?” and concentrate on that. But given that life is so unpredictable, what seemed initially like an inferior option may turn out to be exactly what we need. It’s sensible in many areas of life to leave room for exploring parallel possibilities—if you want to make friends, join several social clubs, not just the one that appears most promising—but it is particularly true in the area of innovation, where a single good idea or new technology can be so valuable. In an uncertain world, we need more than just Plan A; and that means finding safe havens for Plans B, C, D, and beyond.

The Spitfire was a long way down the alphabet from Plan A, not least because the Galapagan isle from which it emerged was populated by some highly unlikely characters. There was Noel Pemberton Billing, a playboy politician most famous as a campaigner against lesbianism. Billing successfully provoked a sensational libel trial in 1918 by accusing the exotic dancer Maud Allan of spreading this “Cult of the Clitoris” and then used the trial to publicize his rather unconventional view that almost 50,000 “perverts” had successfully been blackmailed by German spies into undermining the British war effort.

When not whipping the media into a frenzy about seditious sapphists, Billing was running Supermarine, a ragtag and notoriously disorganized aeronautical engineering company which in 1917 had employed a second unlikely character: a shy but bloody-minded and quite brilliant young engineer by the name of Reginald Mitchell. On his first job, the foreman complained that Mitchell had served him a cup of tea that “tastes like piss.” For the next brew, Mitchell steeped the tea leaves in his own boiling urine. “Bloody good cup of tea, Mitchell” was the response.

No surprise, then, that Mitchell reacted furiously when the large defense engineering company Vickers bought Supermarine and tried to place him under the supervision of the great designer Barnes Wallis—who later became famous as the creator of the bouncing bomb used by the Dambusters. “It’s either him or me!” Mitchell fumed. Whether by good judgement or good fortune, the board of Vickers Aviation decided Barnes Wallis should be moved elsewhere, and Mitchell’s team continued to enjoy Galapagan isolation from the committees of Vickers.

Then there was the most unexpected escape of all. In 1929 and 1930, Mitchell’s planes—the direct ancestors of the Spitfire—held the world record for speed, winning the Schneider Trophy set up to test competing designs. But the government, which was providing much of the funding for these record attempts, decided that they were frivolous in a time of austerity. Sir Hugh Trenchard, marshal of the Royal Air Force at the time, called high-speed planes “freak machines.” Without the development money for the latest world record attempt—and with Henry Cave-Browne-Cave not yet on the scene to pay for an “experiment”—Supermarine was set to abandon the project.

Rescue came from the most unlikely character: Dame Fanny Houston, born in humble circumstances, had become the richest woman in the country after marrying a shipping millionaire and inheriting his fortune. Lady Houston’s eclectic philanthropy knew few bounds: She supported oppressed Christians in Russia, coalminers, and the women’s rights movement. And in 1931 she wrote a check to Supermarine that covered the entire development costs of the Spitfire’s predecessor, the S6. Lady Houston was furious at the government’s lack of support: “My blood boiled in indignation, for I know that every true Briton would rather sell his last shirt than admit that England could not afford to defend herself against all-comers.” The S6 flew at an astonishing speed of 407.5 mph less than three decades after the Wright Brothers launched the Wright Flyer. England’s pride was intact, and so was the Spitfire project. No wonder the historian A.J.P. Taylor later remarked that “the Battle of Britain was won by Chamberlain, or perhaps by Lady Houston.”

The lone furrow ploughed by Mitchell predated by over a decade the establishment of the celebrated “Skunk Works” division of Lockheed. The Skunk Works designed the U-2, the high-altitude spy plane which produced photographs of nuclear missile installations in Cuba; the Blackbird, the fastest plane in the world for the past 35 years; and radar-invisible stealth bombers and fighters. The value of the “skunk works” model—a small, unconventional team of engineers and innovators in a big corporation, deliberately shielded from a nervous corporate hierarchy—has since become more widely appreciated. Mitchell’s team, like the Skunk Works, was closely connected with the latest thinking on aeronautical engineering: Mitchell tested his designs against the world’s best each year in the Schneider Trophy races. But the team was isolated from bureaucratic interference. In a world where the government was the only likely customer, this was no small feat.

Protecting innovators from bureaucrats won’t guarantee results. On the contrary: We can confidently expect that most of the technological creations that stumble out of these Galapagan islands of innovation will prove singularly ill-equipped to thrive in the wider world. But if the occasional Spitfire also results, the failures will be worth it.

Coming tomorrow: What the Spitfire can teach us about medical research.

Correction, May 16, 2011: This article originally misspelled Henry Cave-Browne-Cave’s last name.