For more than 200 years, the federal government has regularly taken an immense survey of American business called the Economic Census. Though not as well-known as the decennial census, the big population count in which enumerators tally Americans house to house, it has been conducted at least every five years since 1905, with a gap only during World War II. Its basic measurements of economic activity, like jobs and revenue, are crucially important to companies, policymakers and anyone trying to track the nation’s economic health.

The next Economic Census was supposed to start in January, five years after the previous one, as usual. But earlier this year, the Census Bureau quietly changed its deadline, pushing it back at least six months. The agency told POLITICO that it has not publicly announced the delay but confirmed that aspects of the Economic Census were "re-planned," and the results would be out six months late.

“If the Obama guys had quietly suggested delaying the Economic Census by six months, there’d be holy hell to pay,” said a former high-ranking appointee in the Commerce Department.

According to multiple statistical experts who spoke with Census Bureau officials, the reason was money: The Census needed the money earmarked for the Economic Census to prepare for the 2020 decennial, which Congress has underfunded by hundreds of millions of dollars. In a tight budget environment, the bureau was effectively forced to choose between two of Washington’s most important efforts to collect data on the country. Even if it’s conducted on the new schedule, the delay of the 2017 Economic Census will have negative effects down the line; it leaves outdated baseline numbers in place for policymakers, and creates problems for companies that need to comply. Said another census-watcher of the 2017 survey: “It will always have this asterisk.”

Such asterisks are popping up more and more in the sleepy world of federal statistics. As wonky as it may sound, collecting and publishing information on Americans and U.S. businesses is one of the most important roles of the government: Information provided by Washington helps small businesses decide the next town in which to expand, and determines the destination of more than $400 billion in federal spending each year. The government has no fewer than 128 agencies that collect and disseminate numbers, including 13 whose primary responsibility is statistics. Its surveys cover topics from inflation to oil prices to mink pelt production. As technical and dry as they are, the data overall form the backbone of U.S. economic planning.

“There are very few things you can get a bunch of economists to agree on, but this is it,” said Tom Beers, executive director of the National Association for Business Economics, about the importance of federal statistics.

But today, under pressure from nearly a decade of budget stagnation, the system is nearing a breaking point. Conversations with more than two dozen statistical experts, present and former officials, find agencies sharply cutting back on the scale and ambition of their data-gathering, reducing sample sizes, delaying investments and, in some cases, eliminating surveys altogether—a last-resort measure that forever leaves a hole in what we know about how the nation is changing. The Bureau of Labor Statistics, for instance, which publishes the nation’s monthly jobs report, has seen its funding fall by nearly 10 percent since 2005, after adjusting for inflation. In turn, the agency has scaled back its quarterly survey on employment and wages, which serves as a benchmark for the jobs report; in 2013, it was forced to stop a survey on mass layoffs and another on green jobs. The little-known Bureau of Transportation Statistics, which collects national transportation data, has seen its budget decline by 21 percent and has been unable to collect data on the number of trucks and their use in the United States.

Getty Images

The biggest anxiety looming over this landscape is the 2020 census. Mandated by the Constitution and conducted without fail every 10 years, the census is the most important and expensive project of any kind that the government regularly undertakes. But it’s already well behind schedule. Strapped for funding in the 2016 and 2017 budgets, it has canceled two of the three field tests scheduled for 2018 and pushed back its advertising campaign designed to get people to answer the survey. When the Census Bureau asked for money in the three-month stopgap spending measure that passed in early September, Congress denied those funds.

“The 2020 census is still in significant trouble, budget-wise,” said Terri Ann Lowenthal, former co-director of the Census Project, an organization that tracks the census. “That keeps me up at night.”

The troubles at the census, as well as many other statistical agencies, predate Donald Trump’s presidency; they’re part of a discretionary budget squeezed hard by Congress and the 2013 sequester. But the Trump administration has brought a fresh wave of concern in the form of a president skeptical about the value of government overall, and whose administration doesn’t seem to be concerned about the gaps it could leave. Early anxieties about a “war on data” by the new administration haven’t borne out, but indifference is a risk too: While both George W. Bush and Barack Obama put a great deal of emphasis on the value of data, Trump still hasn’t nominated anyone to run the Census Bureau and the BLS, arguably the two most important statistical agencies.

Beyond the tight budgets and neglect from the White House, a larger risk lurks around the corner, one that threatens the very reliability of statistics: People simply aren’t answering surveys the way they used to. Because of demographic and technological changes, response rates for federal surveys have been falling for years, a trend that shows no sign of reversing. This makes surveys more expensive to conduct—exacerbating the budget problems at agencies that run them—and raises something of an existential threat to what we know about our own country. “It’s a slow-moving train wreck,” said Erica Groshen, who led the BLS from 2013 to 2017.

As data become ever-more important to business and governance, federal statistics are becoming more expensive—and, the fear is, less reliable. There is one bright spot, however: New ideas and techniques are giving the government lots of ways to replace, even improve upon, its collection of information. By using data already collected by the government and purchasing data from the private sector, Washington can provide a fuller, timelier picture of America—from quicker numbers on economic growth to more accurate information about our health. Nancy Potok, chief statistician of the United States, calls this opportunity a “watershed moment,” and many experts said an update is long overdue; the federal statistical system has essentially been unchanged for decades, even as the economy has become globalized and technology has reshaped industries.

“We’re doing about the same set of indicators that we did 20 years ago,” said Robert Groves, who led the Census Bureau from 2009 to 2012 and recently chaired a National Academies panel on improving federal statistics. “Name something in the economy that is done precisely the same way as 20 years ago.”

But for all the promise of so-called Big Data, so far, the agencies have made little progress incorporating it—in large part, experts said, because they simply don’t have the money for the upfront investment it requires. Buying data from companies, testing it and researching alternatives can be expensive. And as statistical agencies are always in search of areas to cut, such research and development is often the first casualty.

The question now is whether Congress will actually realize the threat facing federal statistics and make the necessary investments to modernize the system, using the vast swaths of data unlocked by new technologies to produce more accurate, timely information. Or whether continued neglect will undermine the federal statistical system, dooming one of the America’s great economic advantages.

PERHAPS NO STATISTICAL agency faces as tough a budgetary environment than the Bureau of Labor Statistics, the 2,400-person agency founded in 1884. Every month, it produces two of the most important and closely watched economic statistics: Its 60,000-person survey of American households generates the unemployment rate, and a different survey, which asks people about the price of recent purchases, produces the inflation rate.

For all its importance, though, the BLS budget has flatlined for almost a decade—and sharply declined, if adjusted for inflation. In 2010, Congress funded the agency at $611 million; for 2017, that number was $609 million, an inflation-adjusted decline of more than 10 percent. At the same time, survey response rates have gone down, making it more expensive to produce the same reports. For the survey that estimates the unemployment rate, called the Current Population Survey, response rates have fallen from around 92 percent in 2010 to 87 percent today. That may not seem like much, but for statistics that are watched to the tenth of a percentage point—the unemployment rate is currently 4.2 percent—small changes in response rates can create subtle biases that affect the data, making the results less reliable.

In 2013, faced with a large budget cut of $30 million, about 5 percent of its previous budget, BLS canceled surveys on green jobs and on mass layoffs, and discontinued a program that made it easier to compare international and domestic statistics. The agency got just half of that cut restored in 2014, forcing it to curtail its quarterly survey on employment and wages; BLS also announced it was canceling a price index on exports but reinstated it after facing a backlash from the business community.

With current programs already underfunded, the agency has been unable to invest in new innovative approaches and upgrade its current surveys. For instance, for years, BLS has tried to get money to fund a survey on alternative working arrangements, such as independent contractors like Uber drivers—currently a significant gap in what we know about the job market. BLS last conducted the survey in 2005, before the rise of the so-called gig economy, and policymakers haven’t had up-to-date data since. Even after it gained a congressional champion in Sen. Mark Warner (D-Va.), Congress failed to provide the $1.6 million to run the survey. The agency conducted the survey again last May, by shifting money internally, but with just two data points in a 12-year stretch—and no money for future surveys—it’s tough to get an accurate picture of how the labor market is changing. “I worry that there’s lots of issues like that where we are going to be behind the curve,” said Beers.

Some surveys remain grounded altogether: The last time the agency measured employer-provided job training—a top policy priority for Democrats and Republicans—was 1995. Other surveys are out of date. The BLS is currently trying to update the geographic locations in its inflation estimates and reduce the complexity and time required for a survey on consumer expenditures. But funding shortfalls have forced it to scale back its plans: It has reduced the number of cities in the inflation measure, and the update to the consumer expenditures survey could take 20 years.

THE BLS REPRESNTS an extreme example of the challenges facing the entire statistical system—but it isn’t alone.

At the Bureau of Justice Statistics, response rates for its high-profile national crime victimization survey fell from 90 percent in 2008 to 76 percent today. At the same time, budget cuts have forced the BJS to suspend surveys on public defense functions and indigent defense. The Bureau of Transportation hasn’t had the money to update its survey on truck use since 2002, and the Economic Research Service at the U.S. Department of Agriculture has cut surveys on smaller commodities, like strawberries. “If you are going to drop something, you drop the minor ones,” said Katherine Smith Evans, a former head of the ERS, “even though, to some extent, the information is more valuable to small markets than to large ones.”

Charlie Rothwell, director of the National Center on Health Statistics, a little-known agency within the Centers for Disease Control and Prevention that produces statistics on births, deaths and drug overdoses, said that flat budgets and declining response rates were forcing the NCHS to “innovate around the edges.” Response rates for the NCHS’s big survey on the health of adults and children have dropped from the high 60s to the mid-50s in recent years, forcing the agency to spend more money to achieve a representative sample—money that can’t be used to research and develop new innovative survey techniques and experiment with alternative data. “It’s like driving an Indy car,” said Rothwell. “We keep messing around with the engine, but maybe we ought to replace the engine at some point.”

Of course, a lot of data collected by the government aren’t of broad national importance; the fate of our democracy doesn’t rest on the strawberry or mink count. But it does depend greatly on the big kahuna—the Census Bureau, which dwarfs the rest of the statistical system. Its decennial census is a massive undertaking that requires a huge administrative ramp-up in the three years prior to the national count. After technological problems in the 2010 census led to a $3 billion cost overrun, Congress put the bureau on a tight budget, a big challenge as response rates declined, requiring expensive in-person follow-ups. “The only thing you can do as the country gets more diverse, living arrangements get more complicated—you need more bodies,” said John Thompson, who resigned as director of the Census Bureau in June. “And that costs a lot.”

Typically, Congress dramatically boosts the agency’s funding in the run-up to the decennial so it can hire hundreds of thousands of people and launch dozens of field offices. The Census Bureau did receive an increase this year, but one that was $200 million below its budget request—and, at 7 percent, significantly below the 11 percent increase it received during the ramp-up a decade ago. In turn, the agency has canceled two of the three sites where it intended to hold its 2018 dress rehearsal, a crucial test for the decennial; it also canceled tests of special procedures to count Americans living in rural communities and in areas recovering from natural disasters. Ironically, Thompson said, the tests were intended to save money by trying out new techniques; without the testing, the agency will have to revert to techniques that worked well in 2010 but are more expensive. “If you want to save money on the census, you need to spend money in Year 7, not Years 9 and 10,” said Groves, Thompson’s predecessor.

In the Trump era, experts worry that the most powerful advocate for the Census Bureau and other statistical agencies—the executive branch—is no longer going to play that role. During the Obama administration, the statistical agencies repeatedly asked for funding hikes, only to be denied by Congress. Agency heads defended those figures at hearings on Capitol Hill, passionately advocating for more money. But Trump’s 2018 budget, which includes huge cuts to discretionary spending, proposes cuts to many of the statistical agencies. Its $1.5 billion request for the Census Bureau—even if Congress funds it at that level—falls $350 million short of what the Obama administration projected the agency would need. When Trump officials trek to Capitol Hill, they’ll have to defend these cuts, rather than press to reverse them.

An official in the White House’s Office of Management and Budget defended the budget requests as viable. “We are committed to having an accurate 2020 Census, and are working closely with the Commerce Department to make sure that the resources necessary to assure a successful census will be available,” the official said.

That has left statistical experts especially discouraged and concerned about the future of the federal statistical system. In the past, statistical experts and agency officials could appeal to certain lawmakers who championed the mission. But when I asked who championed the agencies now, I was met with silence. “The BLS is without champions right now,” said Groshen.

Said Beers, “A lot of other functions of the government have well-funded lobbying groups to make sure that a program continues. There’s nothing like that for statistics.”

FOR ALL THE angst about the future of government data-gathering, there’s a lot of excitement as well. The world is awash in data like never before—a shift that offers huge opportunities to update a statistical system that relies on the age-old techniques, like phone surveys and knocking on doors.

Nearly everyone who tracks government statistics believes this means Washington will be able to produce more accurate statistics at a much lower cost. And there are hopes that it will also reduce survey complexity, saving respondents hours of time finding paperwork and attempting to understand complicated questions—but it also comes with some big challenges and risks. The first is the biggest: rethinking how our statistical system operates. Julia Lane, a professor at New York University, has worked with federal agencies on this issue and seen the challenge up close. “You start with masses of data from all sources,” she said, “and you have to figure out how to link those sources, figure out what’s missing, deal with very rapid updates, how to make sense of it, protect confidentiality.”

Getty Images

A big part of the challenge is personnel, and the new kinds of data scientists the government will need to recruit. “You can think of the statistical agencies as being data factories,” said Lane. “You have to retool an entire production line. … It’s like retooling a factory when your machines are actually people.” The problem is that when agencies prioritize resources, programs like training are often first cut, said Potok, the U.S.’ chief statistician.

Some of the data the government needs come from elsewhere in the government—or, theoretically, they could. For instance, instead of asking about a person’s income—a question that causes many respondents to drop off—an agency could get that information directly from the IRS, which records it from federal tax returns. Similarly, unemployment insurance records could provide data on wages.

So far, such data sharing has occurred only sparingly. The statistical agencies aren’t built to use third-party data—even data from the rest of the government—and they don’t have the institutional capacity or money to develop ways to incorporate it. But there are some signs of progress: In one notable example, the Census Bureau and the Bureau of Economic Analysis have tested a new way to estimate retail sales, a monthly report that financial analysts complain is too slow and does not provide enough geographic detail. As Americans increasingly use plastic to purchase clothing and other retail goods, the government has experimented with purchasing credit card data from third-party aggregators, and so far, the data have tracked the retail sales figures pretty closely.

“You don't know what an individual purchased, but in an aggregated way, you can see sales of washing machines in Chicago yesterday,” said Potok, who previously was deputy director of the Census Bureau. “You can produce it faster. It's probably cheaper.”

For skeptics of Big Government, the idea of replacing the retail sales survey with credit card data is a perfect example of how, in many cases, the private sector can track statistical information better than the government; a job that once required the vast network and infrastructure of the federal government can now, through the help of new technologies, easily be completed by private companies. But Potok also warns that commercial data isn’t a “magic solution.” What happens if the third-party aggregator changes its methodology or jacks up the price of the data? There’s real value in producing the data in-house, where agencies must follow a careful set of rules in their surveys, so that users and respondents alike know exactly what is being collected, how it’s going to be used and its reliability. “It’s all very transparent,” said Potok. “When you are buying data from a company, it's not transparent at all. If they change their methods at all, it could skew the numbers and you wouldn't know it.”

Privacy advocates are also concerned that linking administrative data could cause the inadvertent disclosure of respondents’ personal information. “The more you link stuff together and provide that data, you have a greater chance to identify someone,” said Rothwell, head of the National Center for Health Statistics. Privacy was also a top concern for the Commission on Evidenced Based Policymaking, which was created under a 2016 law championed by House Speaker Paul Ryan (R-Wis.) and Sen. Patty Murray (D-Wash.). In a 138-page report released in September, the commission recommended ways to increase the availability and use of data, including strong privacy protections.

“I thought I understood about privacy and confidentiality from my time spent at BLS,” said Katharine Abraham, an economics professor at the University of Maryland who co-chaired the commission. “But I didn’t really understand the set of people who think about these issues, how deeply they feel about them.” One idea popular among people who prioritize expanding access to data—a central clearinghouse for all federal data—was heavily opposed by privacy advocates on the commission who said it would be a perfect target for cyberattack. “In the end, I was convinced it wasn’t a good idea,” Abraham said.

Getty Images

As agencies attempt to link more and more data, these are the types of big, important questions they are going to face, and experts aren’t exactly sure how to answer them. But they do know that the traditional surveys that have been the backbone of federal statistics for over two centuries are not enough in the 21st century. “It’s an existential challenge,” said Lane. “They can’t keep collecting data through surveys. They know that.” Said Potok: “If we don't do something now, we could have a crisis in the future. I don't think we're in a crisis state now.”

The hope, experts agree, is that we are on the verge of something of a statistical revolution, one that will update the current slow, outdated system to meet the challenges in the fast-paced, globalized economy. “We’re still measuring on a monthly cycle—that’s the best we can do—yet there are massive decisions being made on a faster cycle,” said Groves. “We have a statistical system that did quite well for an economy that no longer exists.”

The risk, though, is that budget cuts and declining response rates undermine data quality before agencies can research and incorporate new data sets. So far, we haven’t reached the breaking point—a big mistake in the unemployment rate, for instance—but no one can say for sure that such a crisis won’t hit soon. “When a bridge falls down, then suddenly people realize, wow, we needed that bridge,” said Groves. “It’s just like that. We’ve enjoyed a wonderful run. But this needs attention.”

Danny Vinik is the assistant editor of The Agenda at POLITICO.

Authors: