Justin Fox started it, and Robin Hanson has a good restatement of the puzzle:

The S&P 500 are five hundred big public firms listed on US exchanges. Imagine that you wanted to create a new firm to compete with one of these big established firms. So you wanted to duplicate that firm’s products, employees, buildings, machines, land, trucks, etc. You’d hire away some key employees and copy their business process, at least as much as you could see and were legally allowed to copy. Forty years ago the cost to copy such a firm was about 5/6 of the total stock price of that firm. So 1/6 of that stock price represented the value of things you couldn’t easily copy, like patents, customer goodwill, employee goodwill, regulator favoritism, and hard to see features of company methods and culture. Today it costs only 1/6 of the stock price to copy all a firm’s visible items and features that you can legally copy. So today the other 5/6 of the stock price represents the value of all those things you can’t copy.

Check out his list of hypotheses. Scott Sumner reports:

Here are three reasons that others have pointed to: 1. The growing importance of rents in residential real estate.

2. The vast upsurge in the share of corporate assets that are “intangible.”

3. The huge growth in the complexity of regulation, which favors large firms.

It’s easy enough to see how this discrepancy may have evolved for the tech sector, but for the Starbucks sector of the economy I don’t quite get it. A big boost in monopoly power can create a larger measured role for accounting intangibles, but Starbucks has plenty of competition, just ask Alex. Our biggest monopoly problems are schools and hospitals, which do not play a significant role in the S&P 500.

Another hypothesis — not cited by Sumner or Hanson — is that the difference between book and market value of firms is diverging over time. That increasing residual gets classified as an intangible, but we are underestimating the value of traditional physical capital, and by more as time passes.

Cowen’s second law (“There is a literature on everything”) now enters, and leads us to Beaver and Ryan (pdf), who study biases in book to market value. Accounting conservatism, historical cost, expected positive value projects, and inflation all can contribute to a widening gap between book and market value. They also suggest (published 2000) that overestimations of the return to capital have bearish implications for future returns. It’s an interesting question when the measured and actual means for returns have to catch up with each other, what predictions this eventual catch-up implies, and whether those predictions have come true. How much of the growing gap is a “bias component” vs. a “lag component”? Heady stuff, the follow-up literature is here.

Perhaps most generally, there is Hulten and Hao (pdf):

We find that conventional book value alone explains only 31 percent of the market capitalization of these firms in 2006, and that this increases to 75 percent when our estimates of intangible capital are included.

So some of it really is intangibles, but a big part of the change still may be an accounting residual. Their paper has excellent examples and numbers, but note they focus on R&D intensive corporations, not all corporations, so their results address less of the entire problem than a quick glance might indicate. By the way, all this means the American economy (and others too?) has less leverage than the published numbers might otherwise indicate.

Here is a 552 pp. NBER book on all of these issues, I have not read it but it is on its way in the mail. Try also this Robert E. Hall piece (pdf), he notes a “capital catastrophe” occurred in the mid-1970s, furthermore he considers what rates of capital accumulation might be consistent with a high value for intangible assets. That piece of the puzzle has to fit together too. This excellent Baruch Lev paper (pdf) considers some of the accounting issues, and also how mismeasured intangible assets often end up having their value captured by insiders; that is a kind of rent-seeking explanation. See also his book Intangibles. Don’t forget the papers of Erik Brynjolfsson on intangibles in the tech world, if I recall correctly he shows that the cross-sectoral predictions line up more or less the way you would expect. Here is a splat of further references from scholar.google.com.

I would sum it up this way: measuring intangible values properly shows much of this change in the composition of American corporate assets has been real. But a significant gap remains, and accounting conventions, based on an increasing gap between book and market value, are a primary contender for explaining what is going on. In any case, there remain many underexplored angles to this puzzle.

Addendum: I wish to thank @pmarca for a useful Twitter conversation related to this topic.