There are two overlapping refrains in coverage of the trendiest boroughs of America’s largest city: “Neighborhood reaches new level of twee luxury” and “Real estate prices reach new level of absurdity.” Stories falling into those two categories account for a good chunk of the New York Times, New York magazine, and the New York Observer. And it’s true that parts of Manhattan can feel like a museum preserved for tourists or an upscale outdoor shopping mall, while “Brooklyn” has become a misused byword for moneyed hipster affectation.

Yet recently released census data paints a different portrait. Measured by median income, Manhattan and (especially) Brooklyn are much poorer than you think. Manhattan’s median annual household income is $66,739, while Brooklyn’s is a mere $44,850. Its less fashionable neighbor, Queens, outearns Brooklyn at $54,373 per year. New York City’s most suburban borough, Staten Island, is also its richest, with a median household income of $70,295, while the suburban counties surrounding New York are all richer than any of the boroughs.

Meanwhile, the cost of living is astronomical in Manhattan, where the median monthly rent is $3,100; it’s $2,800 in the gentrifying northwestern quadrant of Brooklyn. So how can so many relatively low-income people still live in these areas?

The answer has to do with the peculiarities of New York’s housing stock, demographics, and history. Here are the main seven factors.

Geography. Although Upper Manhattan is being slowly conquered by yuppies, Harlem and Washington Heights, which has the largest immigrant population of any New York neighborhood, are still far more racially and socio-economically diverse than more rarefied neighborhoods such as Tribeca and Greenwich Village. In East and Central Harlem there are numerous census tracts with median household incomes below $20,000. Likewise, there are a few wealthy slivers of gentrified Brooklyn, such as Park Slope, Cobble Hill, and Dumbo, and some solidly middle-class enclaves in deep Brooklyn such as Bay Ridge and Mill Basin. But vast swaths of central, southern, and eastern Brooklyn are populated by the poor or working class, many of them immigrants. Neighborhoods like East New York, Sunset Park, East Flatbush, and Brownsville are much cheaper than Williamsburg, but they are simply left out of the surveys of Brooklyn real estate that generate misleading headlines about skyrocketing Brooklyn rents—and they are erased from the “Brooklyn” of, say, a New York Times columnist’s imagination.

Rent regulation. As of 2011, according to a report by the Furman Center for Real Estate and Urban Policy at New York University, there were 1,025,214 rent-regulated units—meaning that the annual rent increase is limited by state law—accounting for 47 percent of the city’s total rental housing stock. Of those units, 284,089 are in Manhattan and 306,374 in Brooklyn, accounting for nearly half the rental market in both boroughs. The disparity in Manhattan between market-rate and stabilized rents has never been greater. In Manhattan, the median market-rate tenant makes $100,000 per year and pays $2,625 per month in rent, while the median rent-regulated tenant makes $49,200 and pays $1,295. People don’t easily give up such a good deal: Fully 35 percent of rent-regulated tenants in Manhattan below 96th Street have been in their apartments for more than 20 years. New York also has a dwindling supply of apartments from the Mitchell-Lama program, a mid–20th-century tax incentive to create affordable housing. There are 97 such city-sponsored developments, accounting for more than 44,600 housing units. As Josh Barro of Business Insider observes, “There are lots of cheap apartments in New York. You just can’t get one of them.”

Publicly subsidized housing. New York City has an enormous public housing system: 621,212 residents—or more than 7 percent of all New Yorkers—live in housing projects or in apartments using Section 8 housing vouchers, according to the New York City Housing Authority (NYCHA). Residents in both programs pay no more than 30 percent of their income in rent; average monthly rent is $436. Brooklyn has 100 projects with 58,699 apartments, and Manhattan has 102 projects with 53,890 apartments. The average family annual income in NYCHA projects is $22,994.

History. Whereas housing in upscale suburbs has always been expensive, most urban neighborhoods went through a period of disinvestment, high crime, and “white flight.” Anyone buying in Manhattan or brownstone Brooklyn 40 years ago paid a pittance compared to today. An entire subgenre of gentrification literature consists of stories about people of modest means who bought a building in rougher times and are now sitting on a gold mine—and living in neighborhoods that they could never afford if they were starting out buying or renting today.

Demographics. New York’s foreign-born population has just passed 3 million; nearly 1 million of them live in Brooklyn. Some immigrant enclaves in Manhattan (Chinatown, the Dominican community in Washington Heights, etc.) remain intact, so their census tracts stand out on the map as pockets of relatively modest incomes. Immigrants often afford neighborhoods by squeezing in multigenerational or extended families into small apartments. The average Ecuadorian household in New York City, for example, has two wage earners, versus an average of 1.2 for the city as a whole.

Another key demographic clue to the income figures: New York, especially Manhattan, is a magnet for childless adults of all ages. Nearly half of Manhattan households are made up of just one person. Often, these are people with access to money—their own, or their parents’—beyond their income. Retirees may have a modest retirement income but large savings, especially if they sold a house in the suburbs to buy an apartment in the city. Young, college-educated professionals, aspiring artists, and graduate students may not earn much money, but they can still have more expensive tastes—and more disposable income to fritter away on high rents—than middle-aged parents in the same income bracket. They also require less space than families with children.

Inequality. There is extraordinary income inequality in New York, and even more so in Manhattan, where the mean income of the lowest 20 percent of households was $9,635, compared with $389,007 for the top 20 percent and $799,969 for the top 5 percent. A relatively small number of luxury apartments catering to the super-wealthy bring up average rents, making them much higher than the median. And those eye-popping average rents are the ones the media lazily reports and repeats, without noting that most people actually pay somewhat less.

Living beyond one’s means. Though all housing prices in New York keep rising, the gains of the current recovery have accrued only to the wealthy, while most residents’ incomes remain stagnant. Consequently, people get pushed into paying more than they can afford for their apartment. According to the census, 64 percent of New York City households pay more than 25 percent of their income in rent and utilities, and 44 percent spend more than 35 percent. Homelessness is also at an all-time high, in part because of the growing gap between incomes and rents. “You have this incredible disconnect, where median income has declined and housing costs have gone up,” says Andrew Beveridge, professor of sociology at Queens College. Beveridge attributes Mayor Bill de Blasio’s victory in part to frustration with the lack of affordability. “The average person is not doing as well in New York,” says Beveridge. “Life is harder for them now than it was 10 years ago.”

So next time you’re at a dinner party and somebody complains that New York has become the sole province of rich white people, you can tell him he’s wrong.