From steamy Miami to the thriving cores of cities from New York, San Francisco, Houston and Chicago, swank towers, some of them pencil thin and all richly appointed. This surge in the luxury apartment construction has often been seen as validation of the purported massive shift of population, notably of the retired wealthy, to the inner cities. Indeed with the exception of a brief period right after the Great Recession, there was slightly greater growth in core cities than the suburbs and exurbs. It was said that we were in the midst of a massive “return to the city.”

Yet in reality the movement to the inner core has been much less spectacular than that. Indeed by 2014, growth once again was faster not only in traditional suburbs but also in the exurban areas that were broadly predicted to be the most doomed. At the same time, the fastest city growth, notes economist Jed Kolko, occurred largely in the most “suburbanized” cities, like Phoenix, San Antonio, and San Diego.

One major meme for the luxury developers had to do with well-off retirees—the one domestic population with the money to afford such housing. Newspapers have been crammed with anecdotal stories about this “trend.” Yet analysis of Census trends among seniors shows that the senior percentage share in both the inner core and older suburbs dropped between 2000 and 2010 while growing substantially in the newer suburbs and exurbs. The most recent data show these patterns continue. Since 2010 the senior population in core cities has risen by 621,000 while the numbers in suburbia have surged by 2.6 million.

So who’s buying them? It’s wealthy foreign nationals, largely as investments. In many cases these units are not really residences but pieds-a-terres for the world’s wealthy; in some markets, as many as 60 percent of units are not primary residences. But such sales are susceptible to changes in foreign economies. And today, many of these buyers must contend with slowing economies at home.

Perhaps most damaging has been the decline in many countries, such as Russia, Brazil, Argentina, Canada and some countries of the Middle East, that have been hit hard by the commodity slowdown. Most critical in many markets, particularly in California, has been the economic slow-down in China, now the largest foreign investor in U.S. real estate. In some markets, like Irvine and in Bay Area suburbs, Chinese investors have accounted for upwards of 30 percent of all buyers.

Realtors in Southern California, long a favored destination for Asian investors, report a significant slowdown in investment , particularly along the coast. In some developments, roughly half the Chinese buyers paid cash, often well over $1 million per unit. This in markets where barely 10 to 20 percent of the houses are affordable to the median income family.

Perhaps nowhere in urban America better epitomizes the relationship between foreign capital and high-end real estate than Miami. From 2004 to 2008, Miami enjoyed a massive luxury housing boom, with over 20,000 new units constructed—only to see many go them vacant for years.

The last five years have seen a resurgence once again. As fortunes were being minted, foreign money tended to end up invested in Miami’s luxury towers.

Now this dependence on foreign investment may wreak havoc. Some sources of investment, such as Russia, are drying up as low oil prices dissipate the wealth of that country’s free-spending oligarchs, and now must cope with sanctions over the annexation of the Crimea. In 2013, Russian buyers accounted for 23 percent of Miami luxury condo buyers; in 2014 they accounted for just 7 percent.

But in Miami, the big story has been the Latin Americans. Like the Russians, the major Latin American investors have been hard hit by declining oil and other commodity prices. In 2014, luxury developer Gil Gezer thanked Brazilians, for turning around the Miami high-end condo market; now they seem to be driving the market down. In the fourth quarter of 2015 the number of Miami Beach condo transactions declined nearly 20 percent from a year earlier, while inventory jumped by nearly a third, according to a report from appraisal firm Miller Samuel Inc. The median sales price slipped 6.6 percent. As prices drop and sales slow, more than half a dozen projects have been cancelled.

South Florida may be the ultimate mecca for luxury developers, but it’s hardly alone in facing a high-end property crash. Over the past decade, New York has been inundated with ultra-expensive high-rise real estate. The new towers have affirmed the city’s fundamental attraction to the ultra rich. In Lower Manhattan, 31 towers with over 5,000 apartments are sprouting up, with a median price for condos of $2.43 million, a 70 per cent increase since 2013. The overall Manhattan condo market has shot up “only” 54% to $1.84 million.

As in Florida, some of the problem stems from the retreat of foreign investors. Analyst Sami Karan suggests that rather than a massive demographic evidence of a “return to the city” by the ultra-rich, the luxury surge seems to be mostly a matter of investment strategies that can change more quickly than shifting one’s long-time domicile.

Not surprising then that some developments are being stalled across Manhattan. Property Markets Group and JDS Development Group, developers of the 111 West 57th Street, a 60-unit tower have announced that the once-imminent sales launch at the 60-unit tower would be pushed back for at least a year. In other cases, once ballyhooed conversions of office towers to condos—notably the famous “Chippendale” Sony/ATT building located at 550 Madison Avenue—have been shelved, signaling that some well-fed rats may be deserting the luxury yacht before the fall. The city faces what new analysis by the consultancy Miller Samuel could be a glutted luxury market for the next five years.

But over the past few years, no luxury market has been more over-heated than San Francisco. As occurred in the 1990s, the city’s luxury market has ridden the current tech bubble to unprecedented heights—in the process creating what may be one of the most severe real estate bubbles in the country. In the city proper, the median value of homes has skyrocketed, from $670,000 at the beginning of 2012 to $1.12 million today, a gain of more than 67 percent, according to Zillow.com.

Now there are signs that this boom is about to slow. This stems from two factors—the inability of consumers to afford this housing and the gradual slowdown of the tech bubble. The 87 tech IPOs over the past two years are trading 80 percent below their IPO price, and not surprisingly, venture capitalists are become more wary. Many key firms—Twitter, Hewlett Packard, Yahoo—are all laying workers off.

All this suggests that, as in Miami and New York, San Francisco property owners face stagnant or even declining prices. The market could be further weakened as tech workers and companies head to more affordable markets elsewhere.

Right now the decline in the luxury market has not yet turned into a full-on crash in multi-family housing. But there are some worrisome warning signs, such as rising apartment vacancy rates. Already some markets, such as Houston, seem to be overbuilt, particulary given weakness in the area’s critical energy sector. Other urban cores threatened by overbuilding include such disparate cities as Indianapolis, Raleigh-Durham, Denver and Atlanta. According to the consultancy firm Costar, vacancies in downtown areas are reaching double digits in such attractive markets as Boston, Charlotte and Philadelphia.

In some areas like San Francisco and New York, a rollback of multi-family prices could be beneficial, because high prices are driving young, educated people out to other regions. Since 2010 educated millennials have been headed increasingly to more affordable regions such as Nashville, Orlando, New Orleans, Houston, Dallas-Fort Worth, Pittsburgh, and Columbus. Even Cleveland and California’s exurban Inland Empire, which still has relatively reasonable housing prices, at least by California standards, are growing their millennial workforces faster than places like New York or San Francisco.

Young people may also benefit as units shift from condo to rental. Of course, the weakening market won’t be too good for the investors, developers and landlords, many of whom embraced the “back to the city” mantra with religious zeal and now will have to confront demographic reality.

But other trends suggest that this decline may be more painful than many suspect. We may be entering a phase where we have reached “peak urban millennials” as they head into their 30s, get married and move to the suburbs. The idea that investing in the urban core, and in luxurious density, guarantees a happy result has now lapsed into mythology. It needs to be replaced with something that more accurately effects not what developers hope (or planners decree) but what people need, and can afford.