“Clearly there’s a moral crisis when you see so many people in need of homes and there’s such a glut of vacant ones,” said Rachel Kutler, a leadership organizer with United Workers, which works with the Roundtable.

I went with Tony Simmons, who is homeless, on a tour through a neighborhood the group recently surveyed. Simmons, 53, is wiry and energetic, and talks constantly about vacant homes, the project, and his own history. He served in Operation Desert Storm, after which he spent some time as a drug runner, driving cars filled with drugs between cities on the Eastern Seaboard, partially so he could fund the heroin habit that was taking over his life. Now he’s kicked his habit and thrown himself into activism for the homeless, although he doesn’t yet have his own home.

We start at the church where the group meets before it begins its surveys, and walk down an alley where vacant homes have been turned into a community garden. Crossing a busy street, Simmons points out a deflated helium balloon tied around a telephone pole alongside a photo of a smiling teenager marking a spot where the teen had been shot and killed.

Some of the homes we pass seem almost new; others are sagging, their paint peeling. Simmons has an eye for which ones are too far gone to save and which are uninhabited, even if they don't have plywood on their windows and doors.

“This house is perfect—the windows are intact, the doors are intact, the foundation is strong and there’s no leaking in the basement,” Simmons says in front of one home that has gray and tan bricks on its facade and a bright plywood board on the door. “Yet it’s all boarded-up.”

There are more than 16,000 vacant homes in Baltimore, according to the city. About 30,000 people in the city will experience homelessness over the course of a year, 3,000 on any given night. That’s partially because Maryland, one of the richest states in the union, has some of the least affordable housing in the country. A minimum-wage worker in Maryland would have to work 138 hours a week to afford a two-bedroom unit at fair-market rent, according to the National Low-Income Housing Coalition.

As central parts of Baltimore get trendier, housing there is expected to get more expensive. Earlier this year, Baltimore put forth a plan that would sell 40 percent of the city's affordable-housing stock to private developers so they could rehab it, a move that advocates worry could exacerbate the housing shortage extremely low-income people face. In a report last spring, the Urban Institute estimated that Baltimore had 29 affordable units for every 100 extremely low-income households.

Even in the gutted streets of McElderry Park, where drug dealers still sell on some corners, Simmons and others have noticed green signs posted on the buildings lately. “For Sale: Finished, The Homes at Griffon Station,” they say, even though they are posted on plywood-covered doors and burnt-out houses. Simmons says he suspects the signs are related to nearby John Hopkins University, which is constantly expanding its footprint in the city.

Baltimore closed the waiting list for Section 8 housing vouchers 11 years ago, though it's purging the list and reopening it this year. Adam Schneider, of the Baltimore group Health Care for the Homeless, estimates that 25,000 people are in homes through the Section 8 program, and 25,000 more need the vouchers. It’s not just Baltimore: There are 7 million fewer affordable housing units than are needed in the country, Nan Roman, the president of the National Alliance to End Homelessness, told me.