The U.S. restaurant industry is in a funk. Blame it on lunch.

Americans made 433 million fewer trips to restaurants at lunchtime last year, resulting in roughly $3.2 billion in lost business, according to market-research firm NPD Group Inc. It was the lowest level of lunch traffic in at least four decades.

While that loss in traffic is a 2% decline from 2015, it is a significant one-year drop for an industry that has traditionally relied on lunch and has had little or no growth for a decade.

“I put [restaurant] lunch right up there with fax machines and pay phones,” said Jim Parks, a 55-year-old sales director who used to dine out for lunch nearly every day but found in recent years that he no longer had room for it in his schedule.

Like Mr. Parks, many U.S. workers now see stealing away for an hour at the neighborhood diner in the middle of the day as a luxury. Even the classic “power lunch” is falling out of favor among power brokers.