Jerome Powell, chairman of the U.S. Federal Reserve, speaks during a Senate Banking Committee hearing in Washington, D.C., on Thursday, July 11, 2019.

Federal Reserve Chairman Jerome Powell echoed what most in the homebuilding industry have been complaining about all year. While the nation sinks into a severe housing shortage, homebuilders are struggling against high costs for labor and materials.

In response to questioning from Sen. Tina Smith, D-Minn., Powell said, "What we hear from the homebuilders is that it's a series of factors that are holding them back and challenging affordability."

He pointed to the fact that thousands of skilled workers left the construction industry following the housing crash that began in 2008 and never returned.

"Now you have a shortage of skilled labor, so it's hard to get people on the job, electricians, plumbers, carpenters and other people. No matter what you pay them, just finding people to do that work," said Powell.

Smith responded, "Would you say our immigration policy might have something to do with that?"

Powell replied, "That's what we hear from homebuilders. That's part of it for sure."

Then he added, "Material costs too have gone up, and some of that is tariffs for sure. The homebuilders feel almost like they've been hit by a perfect storm here."

Builders, especially in the south and west, rely on immigrant labor, much of it from Mexico. The Trump administration's tough stance on immigration has hit construction employers hard.

"It's really across the entire labor force, I mean some of the stats are staggering," said Sheryl Palmer, CEO of Arizona-based homebuilder Taylor Morrison, at a recent CNBC Capital Exchange conference. "We've lost productivity, and we haven't brought new kids into the system because it's not this career that folks are aspiring to have, and for all those reasons, we just can't physically solve the problem because we just can't get them [houses] built."

Single-family housing starts have been unimpressive so far this year. In May, they fell 6.4% compared with April and were 12.5% lower compared with May 2018, according to the U.S. Census.

The lack of sufficient home construction, especially at affordable price points, has led to a severe housing shortage. That shortage continues to inflate home prices, keeping would-be first-time buyers on the sidelines. Rents are now rising at a fast clip, as occupancy is unexpectedly high in both apartment buildings and single-family rental properties.

"What I hear from businesses and communities, especially in rural Minnesota but really all over the state, is that the lack of workforce housing, affordable housing for people who have good jobs is actually a real limit on economic growth," said Smith.