The economic recovery is nearly five years old, but many Americans are feeling financially worse off than they were in 2008 according to study released Monday by the PEW Research Center in partnership with USA Today.

According to the study, a smaller percentage of Americans (44%) identify as being middle class than ever before. At the same time, nearly as many Americans (40%) identify as lower-middle or lower class, while the share of Americans who consider themselves upper class has decreased significantly since 2008.

W. Eugene Smith—The LIFE Picture Collection/Getty Images

The results show that, at least when it comes to Americans perceptions, the middle class is a rapidly shrinking demographic. Unfortunately, this isn’t just a case of a gullible public lapping up a media-generated narrative. The data show that the income of the median American has shrunk 8% between 2007 and 2012, and is at the same level today than it was in 1997. The average income for the highest quintile of earners only fell 2% during that time.

If you take wealth rather than just income into account, the situation looks even better for the wealthy. According to research from William R. Emmons and Bryan J. Noeth of the St. Louis Fed, though the American economy has recovered roughly $21 trillion in lost wealth since the depths of the recession, those gains have gone disproportionately to those who own stocks and property hot housing markets like Southern California. And the trend of increasing wealth concentration has been ongoing for decades now. According to the Congressional Research Service, the top 10% of Americans have gone from owning 67% of the country’s total wealth in 1989 to nearly 75% in 2010.

Though labels like “middle class” or “lower class” have always been ill-defined, the fact that so many Americans perceive themselves to be slipping behind in these labels does not bode well for political stability going forward. Many right-leaning commentators have objected to the narrative that wealth and income inequality is increasing in America, arguing that if you take government transfer programs like food stamps or the earned-income tax credit into account, incomes across the spectrum have been rising, just not as quickly as for those at the top.

But empirical data has clearly shown that it’s not absolute, but relative wealth which makes people happy. In other words, the average person is not made happy by how much he has, but how much he perceives himself to have in relation to those around him. Even if it is true that absolute welfare is increasing for the average American, he might still feel like he’s getting the raw end of the deal if he perceives himself falling behind in the race.

Contact us at letters@time.com.