The number of renter households in the top 10th of the income spectrum rose 61 percent over that decade, more than for any other group. So developers are not simply building luxe apartments no one wants to rent.

But they're also responding to the worrisome dynamic that we've made it very, very difficult in many cities to construct market-rate housing that would be affordable to the middle class or modest renters. It's economically challenging for developers to create new apartments the median renter could afford — at about $875 a month — while covering the costs of constructing them.

Height limits, parking requirements and zoning restrictions all push up the cost of construction. So do lengthy design reviews and legal battles with neighborhoods opposed to new development. Developers must also build at the densities communities allow, and in the limited places where they allow higher density. And if a given parcel of land is only zoned for about five stories of apartments, those apartments may have to command $2,500 a month each to make the project profitable.

It has meanwhile grown much harder to build affordable housing with public subsidies (see the ease with which Donald Trump's father once did this in a different era). Public support has dwindled, and developers who now want to tackle such projects often have to weave together multiple complex tax credits and subsidy steams.

Of course, it's true that the vast majority of our rental housing isn't housing that was built in the last few years. And, in theory, as wealthy renters move into these newly built apartments, some of the older housing supply should trickle down to more modest-income renters. From 2003 to 2013, according to the Harvard report, this kind of filtering added about 11 percent more housing to the cheapest rental stock, homes affordable to the one in five renter households who earn less than $15,000 a year.

But most of those filtering gains were offset by other losses: During the same time span, roughly the same amount of low-cost rental housing permanently left the housing stock because of deterioration and demolition. Cheap housing has also been disappearing as developers upgrade old units to "vintage" homes that command higher rents.

In total, the number of low-cost rentals from 2003-2013 increased by about 10 percent. The number of low-income renter households in search of such housing rose, over the same time, by 40 percent. And these families have also largely missed out on the other big shift in the rental market, as single-family homes have been converted into rentals since the housing bust. Those properties are largely out of reach, too.

The end result? It may feel like we're back to building again — and a lot of that new construction is meant for renters, not homeowners — but huge shares of low- and modest-income households are still paying far more in rent than economists think they should: