North Brother Island is a secret hiding in plain sight. Located in New York’s East River, it was once an important part of the city’s infrastructure. In the last 50 years, however, it’s descended into ruin: Buildings have crumbled, vegetation has grown wild, and its primary visitors are now migratory birds. But as photographer Christopher Payne found out in the course of creating his book, North Brother Island: The Last Unknown Place in New York City, the island still has stories to tell.

Payne, a former architect who specializes in photographing what he calls “America’s vanishing architecture and landscape,” became interested in the island while on assignment to document uses of the East River. In 2008, he wrote a proposal to conduct a photography survey of the island, which is usually off-limits to the public. The New York City Department of Parks and Recreation agreed to grant him access on the condition that he only visit between September and March, months when migratory birds—including gulls, herons, cormorants, and egrets—are not nesting there.

Over the next five years, Payne made the 10-minute boat trip from the Bronx’s Barretto Point Park to the island dozens of times, accompanied by parks department staff. At first, Payne had to acclimate himself to the island’s geography, navigating the thick vegetation and discovering the way in which light changed throughout the day. Though the Bronx and Manhattan were within eyesight, Payne found wandering the island alone was a uniquely isolating experience. “Even though visually you have that connection to the city and you can still hear things—I could hear the Mister Softee truck sometimes—there’s still this sense that you’re disconnected,” he said. “Living in New York, everyone craves their own space and isolation once in a while. When you’re on the island you definitely have that. It’s a rare feeling.”

Between the 1880s and the 1930s, North Brother Island was the site of Riverside Hospital, where those suffering from infectious disease were treated in isolation. After World War II, it served as a housing community for returning veterans and their families. In the 1950s and early 1960s, it became a juvenile drug treatment center. While Payne knew the island’s story, he often had trouble finding physical evidence of its past. “It was very hard for me to find the artifacts I expected to find. They really just didn’t exist. Most of the time you’re looking at the shell of a building, and it’s so far gone you can’t even tell what it was used for. It forced me to look closer, to see graffiti on the walls or to look on the floor,” he said. “A lot of it was detective work. It was like trying to invent a life for something, trying to find a shot or a view that suggested what it used to be.”

With time, Payne discovered some of the clues he sought, but he ultimately found he was more attracted to the island’s natural environment than its manmade structures. As the seasons changed, the landscape shifted dramatically, affecting the appearance of the buildings it now dominated. “One thing that struck me was seeing how much nature had reclaimed the island. If you go there and don’t have any idea what the place used to be, you’d assume that’s how it always was, but if you look at the historical photos, you’ll see this campus with manicured streets and lawns. Now it’s a forest,” he said. “I read this book called The World Without Us by Alan Weisman. He described what would happen if people left the planet. He has a chapter on New York City, and what he wrote could have been captions for my photographs.”