Several years ago, my mother was mailed a brochure for Lower Manhattan real estate that featured a familiar-looking townhouse: the building she’d grown up in. When she’d lived there with her mother and sister in the 1960s and ’70s, their apartment was one of six in the building, home to several generations of an Italian-American family. In 2010 or so, it was converted into a four-story, single-family home, and it sold for $14 million in 2011.

That’s not an unusual trajectory for a building in the West Village or in other in-demand New York neighborhoods where new construction hasn’t kept up with demand and developers have abused tenant protections without consequence. According to Streeteasy, the real estate listing company, more than 300 New York buildings are renovated to decrease the number of units each year. They’re concentrated in just a few neighborhoods where developers think there’s demand for bigger, more expensive units—and are adapting properties accordingly.

Central Harlem, the gentrifying area adjacent to Central Park, leads the way: Between 2015 and 2018, 112 Central Harlem buildings “shrank,” taking 831 units off the market. That was nearly enough to counteract all the neighborhood’s new construction, which created 854 new units in that time. The neighborhood appears to be getting bigger, but it’s not.

It’s a reminder that “growth” in cities isn’t always what it seems and that architecture can be an awfully poor proxy for the social structures to which it seems so closely tied. Neighborhoods that appear to be magnets for new people and more apartments may, behind every historic façade, be losing both.

Several other shifts are happening behind the scenes, keeping populations relatively low by historical standards even in the most in-demand neighborhoods. Those include displacement of longtime residents by newcomers; the rise of Airbnb, which has taken thousands of units out of residential supply in virtually every major city; and, most importantly, declining household size—more on this later. In New York, for example, two of the nation’s standard-bearing neighborhoods for gentrification—Williamsburg and the Lower East Side—have yet to recover from the population loss of the 1970s. In the 2010 census, the entire city of Washington sat at just 75 parent of its historic high population mark, reached in 1950.

Chicago’s North Side is one of those places where the population isn’t growing as fast as you might think—and some neighborhoods that appear to be booming have lost residents. Tight zoning laws have clamped down on new construction, while rising home prices have spurred enterprising buyers to renovate the city’s classic two-flats into single-family homes and combine apartments. According to data compiled by the writer Daniel Kay Hertz, between 2000 and 2012, the North Side’s Lincoln Park (poster child of a gentrified neighborhood, population growth flat) lost as many housing units as the South Side’s Englewood (where population declined by 22 percent).

What’s happening now marks the very tail end of a long, long decline in the population density of urban neighborhoods—one that long precedes the current era of gentrification. Between the 1850s and the 1950s, every American city was full of boardinghouses and cheap hotels where newcomers could stay for weeks or for years. The World War II–era housing shortage incentivized owners to further subdivide homes and apartments, especially in black areas where residents had few other options. In 1962 the American Planning Association wrote that the row house “perfectly illustrates conditions of overcrowding.”

Then came white flight, urban renewal, abandonment, and gentrification. Single-room occupancy housing, or SRO, such as boardinghouses, catering to the urban poor, the working class, and new arrivals of all stripes, was one of the main targets of urban liberals hoping to de-slum the inner city. In Homeless, Gerald Daly writes that New York lost 109,000 SRO units between 1971 and 1987. Roughly half the SRO stocks of Los Angeles and Seattle vanished in the same time frame. Richard White has written that Chicago lost 80 percent of its SROs between 1960 and 1980, a decline of about 70,000 units. As early as 1987, the New York Times could write that the “way of the city” was the transition from “elegant single-family townhouses, which turned into middle-class rooming houses, which became slums, which are now chic brownstones.”

Crucially, in many cities, the building stock didn’t change. You’d only know there were fewer people living there if you were counting the doorbells or reading the names in the mailroom. (Only now that the U.S. is experiencing a spike in housing costs with few parallels since the postwar period are we seeing revised policies toward subdividing units. Cities like Boulder, Colorado, are grappling with residents who want to turn detached homes into de facto boardinghouses while California has permitted residents to open new apartments in backyards or garages.)

All that said, this phenomenon was not the main reason that an older neighborhood like the West Village or Lincoln Park would hold far fewer people today than it did 60 years ago, despite looking essentially the same. That’s because of declining household size. The writer Pete Saunders has noted that while Lincoln Park lost 3,000 units (net) between 1950 and 2012, the bulk of its 38,000-person population loss in that time can be explained by shrinking families: The average household is down in that time from 2.92 to 2.01 people. In D.C., the drop from 1950 to 2012 was from 3.2 persons per household to 2.13. The numbers are similar in St. Louis. The architecture may not have changed, but its inhabitants have.

There are plenty of reasons not to look back with nostalgia on that era as a golden age for U.S. cities, chief among them the shameful, institutional racism that pervaded the housing market. But if there are elements of midcentury urbanism that we do want to recapture—busy sidewalks, vibrant neighborhood social institutions, transit ridership—we have to remember that all those buildings were much more full than they are today. Want a city that functions, at street level, like that one did? Unless you’re adding a kid to every family, you had better build some bigger buildings.