Last fall economist Carl Benedikt Frey and information engineer Michael A. Osborne, both at the University of Oxford, published a study estimating the probability that 702 occupations would soon be computerized out of existence. Their findings were startling. Advances in data mining, machine vision, artificial intelligence and other technologies could, they argued, put 47 percent of American jobs at high risk of being automated in the years ahead. Loan officers, tax preparers, cashiers, locomotive engineers, paralegals, roofers, taxi drivers and even animal breeders are all in danger of going the way of the switchboard operator.

Whether or not you buy Frey and Osborne's analysis, it is undeniable that something strange is happening in the U.S. labor market. Since the end of the Great Recession, job creation has not kept up with population growth. Corporate profits have doubled since 2000, yet median household income (adjusted for inflation) dropped from $55,986 to $51,017. At the same time, after-tax corporate profits as a share of gross domestic product increased from around 5 to 11 percent, while compensation of employees as a share of GDP dropped from around 47 to 43 percent. Somehow businesses are making more profit with fewer workers.

Erik Brynjolfsson and Andrew McAfee, both business researchers at the Massachusetts Institute of Technology, call this divergence the “great decoupling.” In their view, presented in their recent book The Second Machine Age, it is a historic shift.

The conventional economic wisdom has long been that as long as productivity is increasing, all is well. Technological innovations foster higher productivity, which leads to higher incomes and greater well-being for all. And for most of the 20th century productivity and incomes did rise in parallel. But in recent decades the two began to diverge. Productivity kept increasing while incomes—which is to say, the welfare of individual workers—stagnated or dropped.

Brynjolfsson and McAfee argue that technological advances are destroying jobs, particularly low-skill jobs, faster than they are creating them. They cite research showing that so-called routine jobs (bank teller, machine operator, dressmaker) began to fade in the 1980s, when computers first made their presence known, but that the rate has accelerated: between 2001 and 2011, 11 percent of routine jobs disappeared.

Plenty of economists disagree, but it is hard to referee this debate, in part because of a lack of data. Our understanding of the relation between technological advances and employment is limited by outdated metrics. At a roundtable discussion on technology and work convened this year by the European Union, the IRL School at Cornell University and the Conference Board (a business research association), a roomful of economists and financiers repeatedly emphasized how many basic economic variables are measured either poorly or not at all. Is productivity declining? Or are we simply measuring it wrong? Experts differ. What kinds of workers are being sidelined, and why? Could they get new jobs with the right retraining? Again, we do not know.

In 2013 Brynjolfsson told Scientific American that the first step in reckoning with the impact of automation on employment is to diagnose it correctly—“to understand why the economy is changing and why people aren't doing as well as they used to.” If productivity is no longer a good proxy for a vigorous economy, then we need a new way to measure economic health. In a 2009 report economists Joseph Stiglitz of Columbia University, Amartya Sen of Harvard University and Jean-Paul Fitoussi of the Paris Institute of Political Studies made a similar case, writing that “the time is ripe for our measurement system to shift emphasis from measuring economic production to measuring people's well-being.” An IRL School report last year called for statistical agencies to capture more and better data on job market churn—data that could help us learn which job losses stem from automation.

Without such data, we will never properly understand how technology is changing the nature of work in the 21st century—and what, if anything, should be done about it. As one participant in this year's roundtable put it, “Even if this is just another industrial revolution, people underestimate how wrenching that is. If it is, what are the changes to the rules of labor markets and businesses that should be made this time? We made a lot last time. What is the elimination of child labor this time? What is the eight-hour workday this time?”

SCIENTIFIC AMERICAN ONLINE

Comment on this article at ScientificAmerican.com/aug2014