He packs a towel into a black cloth shopping bag. It's one of the last things to pack up from his Manitoba Housing apartment.

It's moving day. After four years of living at 170 Hendon Ave. in Charleswood, the man has had enough of living in the troubled housing building and is moving into a private rental suite in a different neighbourhood.

"It's never going to stop, we've been here for four years, it never stopped," he said of the problems in the Manitoba Housing complex.

CBC News is not naming the former resident, or a friend of his who has also raised concerns, because they are concerned for their safety after speaking out.

The former resident says there are major issues with drugs, prostitution and violence in the building.

He said he had been physically assaulted multiple times and witnessed many crimes in his old neighbourhood and thought he was moving into a safer area.

"I lived it once … and I didn't want to see it ever again," he said.

Winnipeg police acknowledge that officers are familiar with the building at 170 Hendon Ave. and say they have had "countless calls" for service there.

In the middle of April, police arrested two men for an aggravated assault that happened there. Police say the two men are known to police and both have gang connections.

When the concerned resident was first awarded a suite in the buff brick-clad building — just a few blocks away from Assiniboine Park — he thought all his problems were over.

"I was moving into a nice complex with Manitoba Housing, I thought it was great," he said. "It was in Charleswood, close to my parents' home and it was a beautiful neighbourhood."

By the first night, the man realized that although it was situated in a nice neighbourhood, near green space, amenities and public transportation, the eight-storey social housing building stands in stark contrast to the surrounding area.

Front doors of the Manitoba Housing apartment building at 170 Hendon Ave. (CBC)

At the end of Hendon Avenue, just a few houses from the apartment complex, some riverfront houses are valued at $800,000. It's a world away from the Manitoba Housing building.

"It's a nightmare," the former resident said. "I just don't understand. A nice beautiful neighbourhood, you have a building that's like, you're almost in an institution."

Manitoba Housing puts a roof over the heads of 32,000 people in the province, spread across almost 18,000 units. In 2015, housing security officers responded to nearly 16,000 calls for service. Over 8,000 of those calls were defined as "security attendances" and more than 2,000 of those were "security incidents." Security incidents include trespassing, vandalism, domestic violence, physical altercations and break and enters.

Housing officials logged 98 security calls for 170 Hendon Ave. last year and 15 of those incidents involved criminal activity.

The former resident spoke to management about the ongoing issues and said he was told repeatedly to write down any complaints he had and that ultimately if he didn't like it, he should go live somewhere else.

The man lives with a learning disability and functions near the level of a fifth-grade student. He has a hard time reading and writing full sentences and is unable to hold down steady employment.

The former resident's friend moved into the complex three years ago. She was placed in the Hendon Avenue building after staying at an emergency shelter in the city. The two have found a two-bedroom private rental place and say they need to get away from the environment in the building.

"It's been nothing but troubles ever since I moved in. Nothing positive came out of this but bad memories," she said.

'I have to leave here'

The man counts on fixed disability support to get by and his friend is also on social assistance. Moving out of subsidized housing means the pair will have to stretch their limited dollars even further to afford a safer home.

"I have to leave here. The management is not doing nothing in this building for four years and I have to make a step and move on. I hate to do it but I have to get away from this lifestyle," the man said.

The department says over the past seven years, Manitoba Housing has made significant improvements in its security operations to address safety and security concerns.

Some of those improvements include the installation of camera systems across many of their larger properties, the provision of on-site security where they have deemed necessary, and a 24-7 on-call Manitoba Housing Security Service.

The city sign entering the Charleswood neighbourhood reads, "Welcome to Charleswood. The suburb beautiful." But for the pair who has moved out, their time there was anything but.