When Harold Jones first bought his home in Canarsie in 1991, he was struck most by the tall trees lining the street. Today, he’s more struck by the “For Sale” signs standing in their place.

The trees went first, and quickly. During Hurricane Sandy in 2012, the waters of the Jamaica Bay inlet that borders Jones’s street surged over the shallow bank, pummeling everything on the block. The signs came later, over several years’ time. Most of Jones’s neighbors tried to stick it out, but, he said, “A lot of people didn’t have the resources to rebuild.” Some homes eventually went into foreclosure, and others were sold. Some residents moved south, while others returned to the Caribbean islands they’d emigrated from. Others left quietly, their whereabouts unknown to Jones.

Since then, Jones has been trying to help keep the old neighborhood alive. After Sandy, he founded one of the area’s few nonprofits, Canarsie Community Development, to help residents prepare for future disasters. But still, he said, fliers arrive weekly in his mailbox from property development companies promising to “buy homes for cash.” Some of his old neighbors are leaving, and they’re being replaced by those who can afford to take on the risk of living in a flood-prone area.

There’s no formal definition of a climate refugee, though the phrase usually refers to people who have been forced from their homes due to climate change—a Pacific Islander retreating from a disappearing shoreline, for instance, or a sub-Saharan African escaping a drought. But there are less extreme examples, like Jones’s neighbors, who stayed put after Sandy but continued to struggle with the financial burden of living in an area with ever-increasing flood risk. It wasn’t the water or winds that pushed them out, but the storm’s economic aftershocks.

In some parts of the country that are threatened by rising seas—like Louisiana or Alaska—government-sponsored relocations of vulnerable communities are already under way. But retreat looks different in cities like New York, which faces more sporadic climate impacts. For these people, “It’s not a planned retreat,” said David Abramson, director of New York University’s research program on population impact and resilience. “It’s an economic retreat.”