More than 40 million American families can't afford their homes, new data shows.

Typically, housing is considered unaffordable when the mortgage or rent consumes at least 30 percent of a household's income. And by the end of 2012, that applied to 35.3 percent families around the U.S., according to a report from The Joint Center for Housing Studies at Harvard University.

Although that figure is down slightly from 2010, it represents a jump of 9 million households from 2002 who are spending a large share of their income on housing. Of those families, meanwhile, nearly 6 million are severely burdened, devoting more than 50 percent of their income to keeping a roof over their head.

Get Breaking News Delivered to Your Inbox

Renters and homeowners are in somewhat different boats.

"For renters, this is indeed a crisis of affordability," the report notes. Since 2000, the percentage of renters spending a large share of their income on housing has steadily increased. As of 2012, with the U.S. economy supposedly in recovery, upwards of half of renters paid more than 50 percent of income on housing.

Share of households paying at least 30 percent of income for housing Constantine

The number of homeowners living in unaffordable housing crested in 2008, at 30.4 percent, up 6 percentage points from 2001. By the end of 2012 it was down to 27 percent. The rate began to fall in 2011, in part because many people were able to refinance their mortgages at much lower interest rates. However, it also dropped because the number of owners with mortgages fell by 2.7 million between 2007 and 2011 as a result of the foreclosure crisis.

The poorer a household, the more likely it is to struggle affording housing. Among those earning less than $15,000 a year -- about the annual earnings at the federal minimum wage) -- 82 percent paid more than 30 percent of income for housing in 2012.

Compared with households living in affordable housing, severely cost-burdened low-income households spend 39 percent less on food each month and 65 percent less on health care. "These cutbacks seriously undermine the basic well-being of low-income households," the report says.

Minorities are especially burdened by housing costs. In 2012, 27 percent of black households, 24 percent of Hispanic households and 21 percent of Asian households paid more than half their income for housing, compared to 14 percent of white households. Between 2002 and 2013, the number of black and Hispanic households paying more than 50 percent of their income for housing rose 5 percent, as opposed to 3 percent for white and Asian households.