NEW YORK (CNNMoney.com) -- Home prices may have fallen this year, but a new study says housing has become more unaffordable. And if interest rates continue to rise, the balance could tip even further.

According to the 2007 State of the Nation's Housing report from the Joint Center for Housing Studies of Harvard University, 17 million of American households in 2005 were putting more than half their income into paying for shelter - a rise of 1.2 million from the prior year, and a jump of 3.2 million from 2001.

"There's an ongoing affordability problem - and it's getting worse," Rachel Drew, a research analyst with the center, said at a conference earlier this week in New York.

Three main factors intersect to affect affordability: mortgage rates, income and prices.

Mortgage rates have generally been a favorable part of the equation. Since the start of 2001, they've ranged from an average of 5.23 percent for a 30-year fixed in June, 2003 to 7.16 percent in June of 2006. Even after the Federal Reserve started raising its rates in June, 2004, mortgage rates stayed low.

Median income, however, has dropped. Real wages fell from 2000 to 2005, according to the report. By 2006 household income was 1 percent below 1999 levels, according to stats from the Current Population Study of the U.S. Census Bureau.

Wages dropped but mortgage rates held steady; affordability shouldn't have suffered too badly if the third part of the equation - housing costs - remained stable.

But they didn't. Single-family home prices skyrocketed, from an inflation-adjusted median of $154,563 in 2000 to $221,900 in 2006, according to the National Association of Realtors (NAR), for an increase of 46 percent.

According to Drew, the affordability decline trend cuts across all incomes. The bottom 25 percent of earners have been hit hardest, but even among the top quarter, the number of households that devote more than half their incomes to housing costs has increased.

That trend accelerated through the housing boom. By 2006, homeowners paid a median of 25.4 percent of their incomes to mortgage payments alone, according to the Harvard study, up from 18.9 percent in 2003.

Housing prices have fallen but the drop has been modest so far; the Mortgage Bankers Association (MBA) forecast a total nominal median home price decrease of between 1 percent and 2 percent this year, hardly a huge savings for buyers.

Not only that, but a drop in housing prices could be wiped out by interest-rate hikes. Freddie Mac, a government-sponsored mortgage-loan buyer, reported that its national average for a 30-year fixed rate hit 5.53 percent last week, a 10-month high. The agency's next mortgage rate report is due Thursday with higher rates expected.

Mortgage rates are benchmarked to bond yields, which have risen this week - the 10-year Treasury hit 5.32 Wednesday for a five-year high.

If the trend continues, analysts, like Doug Duncan of the MBA, have said they expect to see 30-year fixeds topping out near 7 percent by the end of the year, which is 0.85 percent above a May 10th low.

The difference means monthly mortgage payments would be 9 percent higher; it's like adding almost a tenth to the cost of a home, which more than offsets the slight price declines.