It’s not just the poorest city-dwellers who are feeling the rent pressure. As prices rise, even those who make median incomes are finding that their rent eats away at a more significant portion of their pay than it once did for those in the middle class. It’s also not just the Millennial crowd: This problem is also evident across different age groups, including Gen X and Boomers who never left the rental market, or find themselves back in it after the housing crash.

A big part of the problem is that fewer households are making the transition from renting to owning, which means more competition for limited inventory—driving rental prices up. Renters who would previously be able to qualify for mortgages are either finding that mortgage lenders are still super strict post-recession, or that there simply aren’t many homes in their price range—or both. “In normal times when homeownership was achievable you could get a starter home for between $150,000 to $250,000,” says Andrew Jakabovics, a senior director at Enterprise Community Partners, a nonprofit that focuses on affordable housing. “That segment of the market is basically dead.”

Jessica Rinaldi / Reuters

So instead, households with higher incomes and dreams of white picket fences remain in the rental market. Those households take up available units in the mid-to-high price ranges, for which they can afford to pay a premium. In fact, renters with incomes that top $75,000 are among the fastest growing group in the market, says Chris Herbert, the managing director of the JCHS. “Developers will be drawn to build the houses that provide the highest returns,” he says. That means not enough new apartments are affordable apartments that can accommodate low- and middle-income residents. Instead, high-priced luxury units get built first, pushing rents up and middle and low-income earners into apartments that are more expensive than they can afford. Sometimes this means pricing them out of cities altogether.

Pricey rent might not seem like that big of a deal, but it is. Housing accounts for the single largest expenditure for the bottom 90 percent of earners— which is pretty much everyone. While you may be able to scrape and skimp in order to decrease other bills like groceries, utilities, or extras like entertainment, housings costs are generally fixed—at least for a period of time. In other words: The more you’re burning on rent, the less there is for everything else.

And for the 11.25 million households that are considered the most burdened by housing costs—those spending more than 50 percent of their income—they aren’t just spending slightly more than half their income. Jakabovics says that the median of that group is well over the line, and what they spent on rent is closer to 70 percent of their income.

While the strain is awful for individuals and families, it’s actually also pretty bad for the economy. “We know that low-income households that are cost burdened spend considerably less on food and other essentials than affordably-housed folks in the same income bracket,” Jakabovics says. He estimates that even helping those who are most overburdened by the rent to decrease their housing expenses to only 50 percent of their income would put $56 billion back in their pockets. In turn, these households would put that money back into local economies.

But until housing policies change—allowing for more affordable housing to be built more rapidly, or mortgage companies loosen the tight reins placed on credit in the wake of the crisis—rental prices will continue to push upward, and those who can afford the increases least, will suffer most.