Illustration by Christoph Niemann

We live in the age of the startup. It’s never been easier to build a product and start a company. And, thanks to the boom in angel investing and crowdfunding, it’s never been easier for startups to raise money. The analytics firm CB Insights logged more than seventeen hundred seed-investment deals in the U.S. tech industry in 2012, more than three times the number from three years earlier. But there’s a catch: starting a company may be easier, but making it a success isn’t. Competition is fierce, profits are scarce, and venture capitalists aren’t generous when it comes to later stages of funding. As Gideon Lewis-Kraus shows in “No Exit,” a new Kindle Single about startup culture, the life of a new company is often brutish and short. Though we may be seeing a “Cambrian explosion” of new companies, as The Economist recently put it, there’s a mass extinction going on, too.

The fact that most new businesses fail is hardly a secret. So why are so many people gambling on ventures that are likely to end badly? A traditional answer is that entrepreneurs are just more comfortable taking risks than the rest of us. The eighteenth-century Irish-French economist Richard Cantillon, who coined the term “entrepreneur,” defined it as a “bearer of risk.” And in 1921 the economist Frank Knight argued that the function of entrepreneurs was to “specialize in risk-taking.” Yet studies of entrepreneurs find that, in general, they’re as risk-averse as everyone else. Only when it comes to starting a business are they daring. And that’s because the fundamental characteristic of entrepreneurs isn’t risk-seeking; it’s self-confidence. A 1997 study in the Journal of Business Venturing found that entrepreneurs are overconfident about their ability to prevent bad outcomes. They’re also overconfident about the prospects of their business. A 1988 study in the same journal of some three thousand entrepreneurs found that eighty-one per cent thought their businesses had at least a seventy-per-cent chance of success, and a third thought there was no chance they would fail—numbers that bear no relation to reality. A recent paper called “Living Forever” notes that entrepreneurs are more likely than other people to overestimate their life spans.

Entrepreneurs may recognize that, in general, starting a business is risky. They just believe that their innate skills will win out. David S. Rose, a serial entrepreneur turned angel investor, and the author of the new book “Angel Investing,” told me, “You have to have an unreasonable level of confidence as an entrepreneur, or you’ll never get started.” This helps entrepreneurs keep going in what’s often a bleak and frightening endeavor. “Starting a company is extraordinarily difficult, even agonizing,” Rose said. “You need self-confidence and ego to get through it.” At the same time, the fact that so many entrepreneurs are convinced that they will succeed makes success less likely, by swelling the ranks of competitors. This dynamic was made famous by the economist H. Scott Gordon: in a 1954 essay, he noted that, because fishermen were “incurably optimistic” about their abilities to bring in a big catch, there were always too many fishermen working in the ocean, which, in turn, made it harder for them to earn a living.

In the startup world, endemic optimism is amplified by other factors. The ease of developing a product and getting seed money gives entrepreneurs a lot of positive feedback early on. The rise of “accelerators” like Y Combinator—which provide funding and also mentoring and networking services (a kind of boot camp for entrepreneurs)—has made building a business seem less risky. On top of this, there’s a widespread tendency to treat failure as a badge of honor: “Fail fast, fail often” is a familiar mantra in Silicon Valley. There’s now a regular FailCon, where people come to hear other entrepreneurs tell about the hard times they endured and about how starting a business and failing actually makes you more likely to succeed in the future. It’s a comforting message, but the evidence suggests that past failure really just predicts future failure. A 2009 study of venture-backed firms found that entrepreneurs who had failed in the past were not much more likely to succeed in new ventures than first-time entrepreneurs were—some eighty per cent of those who had failed before failed again. A later study of more than eight thousand German ventures came to an even grimmer conclusion: founders who had previously failed were more likely to fail than novices.

You might suppose that entrepreneurs would be better off curbing their optimism. But we’ve built a whole system on unrealistic expectations. Because we don’t know how to identify good companies in advance, investors end up funding lots of them in the hope that a few will hit it big. As a character on the new HBO series “Silicon Valley” says of a V.C., “You know how sea turtles have, like, a shit ton of babies because most of them die on the way down to the water? Peter just wants to make sure his money makes it to the ocean.” The economy has come to rely on this Darwinian process to drive innovation. “Overconfidence means that many more companies start up than will ever succeed,” Brian Wu, a professor of strategy at the University of Michigan, told me. “That’s unfortunate for individual companies. The paradox is that it’s really beneficial for society.” In the delusions of entrepreneurs are the seeds of technological progress. ♦