Last week, someone leaked a copy of a New York Times report on innovation that is shockingly good. It clearly identified the major problems facing the newspaper in an increasingly digital world, and it made some smart recommendations for transforming the Times into a "digital first" publication. If the Times were to follow them, it would have a large and positive impact on the paper. And the fact that the team behind the report was led by Arthur Gregg Sulzberger, the heir to the Sulzberger family that controls the paper, ensures that it will be taken seriously.

Nevertheless, translating the report's recommendations into action will be very difficult. As Ezra Klein points out, the report includes an excellent summary of Clay Christensen's concept of disruption. The report correctly observes that publications like Buzzfeed are posing exactly the kind of disruptive threat Christensen wrote about. But what the report doesn't mention is the sobering conclusion of Christensen's research: companies faced with disruptive threats almost never manage to handle them gracefully.

Customers in control

Why is it so hard to deal with disruptive threats? The reason, in a nutshell, is this graph from the Times report:

After almost two decades of publishing on the web, the New York Times still earns less than a quarter of its revenue from its website. The Times report talks a lot about becoming a "digital-first newsroom," but financially speaking, the Times is still a print-first publication. That's going to be true for several more years at least.

This matters because, as Christensen emphasized, businesses are effectively hostages to their own customers. Businesses become successful by relentlessly focusing on their customers' needs. They develop systems for ensuring that every part of the business is focused on things that will maximize customer satisfaction and — accordingly — revenue.

Over time, the processes and attitudes necessary to succeed in a particular industry become embedded in an organization's DNA. People who demonstrate a knack for serving that industry's customers get respect and promotions. New employees learn attitudes and procedures that help the organization satisfy existing customers and cultivate new ones.

The innovator's dilemma

Most of the time, this cultural DNA is a huge strength. But it can become a liability when a company is threatened by disruptive innovation. Disruptive companies develop simpler, cheaper products that initially serve a different audience than the core customers of incumbent companies. (BuzzFeed, for example, started out with cat videos and quizzes before it started publishing "serious" news.) And crucially, profit margins are usually lower in the new market than they were in the old.

The processes and attitudes necessary to succeed in a particular industry become embedded in an organization's DNA.

Changes that make a news organization better at serving digital readers and advertisers almost inevitably means serving print readers less effectively. And when the leaders of a newspaper like the New York Times propose changes designed to make the organization more web-savvy, the cultural DNA that has served the company well so long becomes an obstacle. After spending decades working to make their organization better at serving print customers, they naturally resist changes that will make them worse at serving those same customers. And in many cases, they have the financial numbers on their side. Web-native publications like BuzzFeed, Business Insider, and the Huffington Post are growing rapidly, but none of them have revenues anywhere close to those of the New York Times.

A good example of this principle in action is the New York Times paywall. None of the company's digital-native competitors have paywalls, and for good reason: to succeed online, you want as many readers as possible. It's hard to be sure, but it seems likely that the paywall is a factor in the slow growth of the paper's web traffic. And while the paywall is lucrative in the short run, the Times is likely forgoing higher advertising revenues it could earn with a larger readership.

The primary purpose of the paywall isn't to generate revenue, it's to prevent people from canceling their print subscriptions

But the primary purpose of the Times paywall isn't to generate revenue, it's to prevent people from canceling their print subscriptions. That's why the company actually charges less for a print subscription with digital access included than a stand-alone digital subscription. (For example, a Monday through Friday print subscription is $7.20 per week, and comes with a digital package that costs $8.75 per week when purchased alone.) The company is — rationally — choosing to hobble its web-based product to avoid cannibalizing its lucrative print business.

Clay Christensen's The Innovator's Dilemma is full of examples like this. For example, in the 1980s DEC was a successful provider of expensive computers being disrupted by much cheaper PCs from Apple, IBM, Compaq, and others. DEC clearly understood the competitive threat from the PC. The company tried four times to respond with cheaper PCs of their own. But each time, the company found it too difficult for a company used to the high profit margins of more powerful computers to make money in the highly competitive PC business. So none of DEC's PCs took off, and by the 1990s DEC had become a minor player in the computing industry.

The lessons of history

This is a story that has played out over and over again. Companies with brilliant executives who clearly understood that they were facing disruptive threats still failed to survive them. It's happened in industries as diverse as steel, hard drives, and steam shovels. Many of these companies tried hard to adapt to changing circumstances and introduce new products that took advantage of new technologies. But they were hobbled by the fact that it's inherently difficult to transform an organization built to serve customers in a high-margin business into an organization that can survive in a lower-margin business.

As I've written before, a newspaper is a large, bureaucratic institution whose every department is carefully crafted to maximize its profitability as a publisher of newspapers. It owns printing presses, warehouses, delivery trucks, newspaper boxes, and other assets that are totally useless for publishing a website. It has a large staff of paper carriers, sales representatives, typesetters, customer service representatives and so forth who would have to be retrained or (more likely) laid off. And most importantly, it has an institutional culture that’s fundamentally about publishing newspapers. The New York Times can’t stop printing newspapers any more than DEC could have just stopped making the high-margin computers that made it successful. Doing so would have meant discarding the vast majority of what made the company valuable.

When executives clearly understand that they're facing a disruptive threat and work hard to address the problem, they still fail most of the time

And as long as the New York Times makes the majority of its income from its newspapers, that will exert an inexorable pull on the culture of the newsroom. Reporters and editors will resist changes that undermine the quality of the newspaper by pointing out — correctly — that print subscribers pay the majority of the bills. Executives on the business side will resist reader-friendly changes to the website out of fear that they will undermine demand for the print product.

The bottom line is that even when executives clearly understand that they're facing a disruptive threat and work hard to address the problem, they still fail most of the time. This part of Christensen's message tends not to be widely emphasized because Christensen himself downplays it. In the introduction to the Innovator's Dilemma, he wrote that "colleagues who have read my academic papers reporting the findings recounted in chapters 1 through 4 were struck by their near-fatalism." But he then goes on to reassure readers that "there are, in fact, sensible ways to deal effectively with this challenge."

Yet Christensen never really delivers on this promise. He offers some ideas for coping with disruptive innovation, but he finds only a handful of examples of companies that have (narrowly) escaped bankruptcy, compared with many companies that simply went broke.

It's not surprising that Christensen chose not to emphasize this aspect of his research. He was, after all, writing a book he hoped business executives would buy, and a book that explains how to save your business is going to be more popular than a book about how your business is inevitably doomed. But the reality is that most businesses threatened by disruptive innovation don't survive. So even if the senior leadership of the Times accepts the findings of the Times innovation report, they're going to find it a huge challenge to make the kind of dramatic changes that will be required for the Times to master the web.