I’ve been thinking a lot about entrepreneurship and innovation lately. First, I gave a talk about innovation, based on Weird Ideas That Work, to a group of about 50 Korean executives from SK Telecom, who were visiting Silicon Valley to learn about how entrepreneurship and innovation works around here. They were given a very high-end experience, spending a day at IDEO learning about their creative process, hearing from CEO Tim Brown and the amazing Diego Rodriguez, getting a speech from innovation guru Clay Christensen, and after my talk, getting on the bus for lunch at Google and a talk from Marissa Mayer. I also have been thinking about entrepreneurship because, as you can see from my last post, I spent a fair amount of time poking around the updated version of the Stanford Technology Ventures Program’s new Educator’s Corner.

As I was preparing – and giving – my weird ideas talk to the SK Telecom executives, I kept thinking, after spending 20+ years in the Stanford Engineering School and studying innovation for a long time, about which single lesson matters most, at least given my biased view and experience. Weird Ideas That Work came out in 2002, but it was based on a talk that I’ve been giving – and tinkering with -- since about 1995. It has 12 weird ideas for sparking innovation. But I’ve come to believe that one of these weird ideas is more important, more empirically valid, and more troubling than the rest:

Step 1: Decide to Do Something the Will Probably Fail

Step 2: Then Convince Yourself and Everyone Else That Success is Certain

Consider the evidence. Most new ideas fail. Most new companies fail, most products fail, and most new technologies fail. Darwinian concepts apply to innovation, as hundreds of careful studies show. The only way to do something that will probably succeed is to replicate the past, especially to make your own future as perfect a replication as the past as possible. When Toyota makes another Camry, McDonald’s sells another Big Mac, or P&G makes and sells another box of Tide, the odds of success are pretty high.

Once you enter the world of innovation, where you are doing something new, despite all the hype, all those consultants with sure fire methods, the fact is that – even if you have Kleiner Perkins funding, an experienced CEO, Wilson Sonsini as your law firm, and a bevy of hot young Stanford engineers developing your product -- odds are you will fail. Those are the hard facts, and although most innovators believe that they are better than the rest, that the odds don’t apply to them, that is a delusion. Google and its founders are worshipped, but at the time they were was funded, local venture capitalists seemed to be just as excited about the Segway (which still may make it, who knows), Zaplet (I think they dropped over 100 million on that one), Guru (another 60 million or so), and the infamous Webvan (over 800 million!).

The investors at top venture capitalist firms and the people who run R&D at places like 3M, P&G, and BMW are very smart people and have a great overall internal rate of return, but as they know so well, in the world of new ideas, they must accept a high failure rate. They also know – and sometimes even admit -- that so-called experts have a terrible record when it comes to predicting which new ideas will survive and which will not. Stanford’s James March, perhaps the most renowned living organizational theorist, summarized this state of affairs elegantly (this quote is in the last chapter of Weird Ideas That Work):

Most fantasies lead us astray, and most of the consequences of imagination for individuals and individual organizations are disastrous. Most deviants end up on the scrap pile of failed mutations, not as heroes of organizational transformation. . . . There is, as a result, much that can be viewed as unjust in a system that induces imagination among individuals and individual organizations in order to allow a larger system to choose among alternative experiments. By glorifying imagination, we entice the innocent into unwitting self-destruction (or if you prefer, altruism).

Yet, the fact remains that, if you are in the innovation business, developing new products, compounds, or services, starting or funding new companies, you don’t make a cent if you don’t place your bets on something or someone: Nothing ventured, nothing gained, is both a cliché and a dangerous half-truth. It is a cliché because, after all, those VC’s wouldn’t have those nice new houses and airplanes courtesy of Google if they hadn’t made their bets. And it is a dangerous half-truth because, well, nothing ventured, nothing lost (think Webvan).

Once you put your money down on something (that will probably fail), that is when things get really weird. Sure, there are things you can do to increase the odds of success, work hard, find smart people, and so on. BUT if I was going to pick one thing (based on the evidence), I’d vote for irrational optimism, convincing yourself and everyone else that success is certain.

Why? There is a huge literature – more than 500 studies out there now – on the self-fulfilling prophecy: If you believe that great things will happen, the odds of success go up, if you believe that bad things will happen, the odds of failure do to. Sociologist Robert Merton wrote classic article on the subject in 1948, and of course, this notion goes back to the story of Pygmalion. And much the same story emerges from medical research on placebo effects, that sugar pills often work just as well as “real medicine” because people have irrational faith.

So, if you look at the innovation and entrepreneurship game from a portfolio perspectives, this means that – although their failure rate will remain plenty high – investors will do better over the long hail if they bet on persuasive optimists. Thomas Edison fits this description perfectly, so does Francis Ford Coppola, Steve Jobs (of the famous reality distortion field), and Burt Rutan. Rutan is the designer of the Voyager, the first plane to fly non-stop around the world, and more recently, won the 10 million dollar X-Prize with two flights by his Spaceship One -- to demonstrate the possible viability of “space tourism.” When so called experts told him that Voyager would never make it, he told his team that “Confidence in nonsense is required.”

The problem with such success stories, however, is that we tell and remember them, and we don’t tell and forget all the failures, all those optimists who go from failed idea to failed new idea. And that is why, as March’s quote hints, there is a true ethical dilemma in the world of innovation. If you are on top of portfolio, being optimistic and funding optimists, increases your hit rate, but as March says, for the typical person, project, or company in the portfolio, the effect is to “entice the innocent into unwitting self-destruction (or if you prefer, altruism).”

There is also a further twist that I’ll talk about in a future post: The same optimism that increases the odds of success also can lead to escalating commitment to a failing course of action. That is why banks have one group to hands-out loans and another to pull the plug. And it is why the lead venture capitalist on a deal often remains behind a company 100% until the moment that his or her partners intervene and terminate the investment. The overly optimistic backers of the adventure will often keep throwing good money after bad because they have too much invested to quit, even though, as they say, sunk costs are sunk and shouldn’t affect decisions. The result is thaqt cooler and more detached folks are enlisted to kill investments that seem doomed.

In short, if your goal is to have the highest internal rate of return for a portfolio of companies or the highest success rate for set of development projects, than a strong empirical case can be made for “Decide to Do Something the Will Probably Fail, Then Convince Yourself and Everyone Else That Success is Certain.” Yet all those sticky and difficult ethical problems remain, and I don’t quite know what to do about them. Think about it, if most entrepreneurs came to grips with how bad the odds against them are, they might be less likely to be “enticed” to start new companies, and thus avoid the pain and expense of failure. But there would be negative effects as well, fewer new ideas, less innovation, less cool new stuff. And if only rational, well-informed pessimists that elected to innovate, to do something new stuff, their failure rate would be even higher.

I once teased a local venture capitalist that, if he gave the entrepreneurs his firm invested in true informed consent, they would have them sign a document that said something like “I understand that, even though I am accepting this money, the odds are only 5% that my company will succeed, and that even if it does succeed, the odds are over 50% that the investors will remove me from the firm." It doesn’t sound like much fun, does it?