ST. PETERSBURG — The discussion became heated as condo owners living within strolling distance of downtown restaurants and museums interrogated both the developer of a 12-story affordable housing building that will rise on their doorsteps and the social services agency whose clients will make their home there.

The condo owners, most with townhomes along Fourth Avenue S between Sixth and Eighth streets said they are worried about what the $22 million project and its residents will mean for their safety, property values and quality of life. And they talked about the misery they are already enduring, including vibrations from the construction that have been disruptive and destructive.

INTERESTING READ:The 53 jobs in Florida that pay more than $100,000

Developer McCormack Baron Salazar, together with the apartment management company and the agency that will provide services to renters in 33 of the 65 units of the new tower, struggled to soothe their fears.

Gary MacMath, president and CEO of Boley, the nonprofit agency that serves the homeless and mentally disabled and will have a case manager at the new Delmar Terrace apartments at 745 Delmar Terrace S, walked out of the meeting at the Hilton St. Petersburg Bayfront.

Boley got the brunt of the anger that night. The agency owns a small building in the neighborhood, on Seventh Street S. Those who live in the nearby condos say Boley residents harass them by panhandling and even show up at their homes. They add that they've witnessed drug deals and that there's been a SWAT team at the facility.

Jacob Unger has owned a two-story Delmar Villas townhome, behind the land where the apartment tower is being built, for two and a half years.

Boley can't control the people who live in their small facility now, he said. "Now they're going to add a bunch of unwanted people, drug users, people in and out of jail," he said. "I am concerned about the property values being diminished, as well as the safety."

RELATED: Where can you earn nearly $120,000 and still qualify for affordable housing

James Ghelarducci said he became so concerned that he moved out of the neighborhood two months ago.

"I have the closest place to the Boley House," he said, of the townhome he now rents to tenants. "I'm getting married within about four months and in preparation for the marriage, I didn't want her to deal with drug addicts, screaming at the moon, crazy people talking to themselves. Now they are going to exacerbate the same problem."

MacMath said the facility, called the Marconi, has 10 apartments for chronically homeless individuals and families. "A lot of them are unable to work," he said, adding that most depend on Social Security or veterans' benefits.

The apartments have been there for 25 years, MacMath said. "When we bought it, that neighborhood was in a very bad shape. We were the first in that neighborhood to start renovation and revitalization," he said. "All of these $300,000 townhomes and condos came in with us there."

Nevertheless, Boley has responded to complaints.

"As far as the panhandling, that was one tenant who no longer lives there. We evicted three tenants, and we fixed up the property. We will try to be more diligent about who we put in the home, but you can't just throw people out," he said. "We just can't please that neighborhood group, and we never will."

Arnold and Lisa Zweben and their daughter have lived in their townhouse for three years.

"It's not the fact that Boley is there, it's the fact that they're bringing a different element into our neighborhood now. And we shouldn't feel threatened living in our own home," Lisa Zweben said.

"There are only 10 units at the Marconi and we are literally being terrorized. We are terrified at the thought of adding 33 more units of the same element. That is our concern. ...We have drugs. Our property has been vandalized repeatedly. We've had vehicles broken into. We had to get a new mailbox system to secure our packages. We've had to get our walking gates retrofitted with mesh and spikes welded on to the top. Basically, we're fortifying ourselves."

Zweben said the condo owners wonder why that particular piece of land was chosen for the project.

"We are almost all townhomes, up and down this street. Why did they then choose to put a 12-story, low-income building in the middle of a full residential townhome community?"

MacMath said Boley was approached by the developer, who wanted to apply for low-income tax credits through the Florida Housing Finance Corporation.

"Part of the application is how you're going to be providing services," he said, adding that the clients the agency will serve would typically be "homeless folks that are newly homeless that have jobs and can pay rent."

"Typically, what we will be doing is support them to maintain their housing," he said. Specific needs could be jobs through one of Boley's job placement programs, literacy training, educational and volunteer opportunities and childcare.

"We went out and we found the land, which was just sitting there," MacMath said.

Kathryn Driver, executive director of the Housing Finance Authority of Pinellas County, said the authority bought the land for $1,050,000 with Penny for Pinellas funds.

"The land is held in the community land trust. The county owns the land, and we lease the land to the developer. We've entered into a ground lease with the developer that they are going to keep those units that are being set aside for affordable housing for 99 years."

Joshua Johnson, St. Petersburg's housing and community development director, said Boley is recognized as one of the best affordable housing providers in the state. "When they have a project, we look to help them, because we know it's going to be done right," he said.

Zweben said neighbors have no problem with affordable housing.

"We're against the type of criminal element that Boley is bringing in," she said. "I think all the people that were at the meeting were satisfied with the management company and their process for bringing in people. We are not against anyone who has fallen on hard times. We are against the mentally ill who are being sent over here without supervision."

Neighbors are also upset that they did not learn about the building until January, when the developer requested a sidewalk variance. After they received a notice about the variance from the city, they scrambled to learn what it meant.

Romano De Simone and his wife, Donna, are new to the neighborhood.

"I just bought this place and moved in at the end of February. We had no idea this was going up. For us, it's more the size and scope of it. It's literally nine and half feet from our back entrance. ... It's just concerning that there's a giant 12-story building squeezed onto such a small piece of land that we feel that the city is just forcing on us," he said.

"We just got married. This is our first property and we were very excited to move in, and now we've got this. We feel blindsided."

Contact Waveney Ann Moore at wmoore@tampabay.com or (727) 892-2283. Follow @wmooretimes