“How can you argue this garden is a slum?” Mr. Siegel said. “This issue is not just about Community Board 2 and Elizabeth Street Garden. If they can take your space away in Little Italy, they can take it away in your neighborhood, too.”

Critics argue there are other green spaces just two blocks away from the garden, like the Liz Christy Community Garden and the expansive Sara D. Roosevelt Park. And the well-funded campaign to save the garden, by residents of a predominantly white and gentrified neighborhood, has been portrayed by some as a classic case of the not-in-my-backyard, or Nimby, attitude.

“We are in a crisis of housing and I don’t think that the desires of a small group of collectively well-off people, some of whom own homes, can really determine the future for a desperate need here,” said Kathleen Webster, 65, a community gardener and Little Italy resident. “Everyone keeps saying build it on Hudson Street.”

Since 2014, only 93 units of affordable housing have been built in Community Board 2, which also includes Greenwich Village and the West Village.

Joseph Reiver, who has ambitious plans to build a greenhouse and community compost in the garden, denied that characterization.

“We shouldn’t be pitting senior affordable housing against community green spaces,” he said. “We don’t have to divide the community.”