Sydney's Lord Mayor and NSW opposition members joined hundreds of protesters in Circular Quay, demanding one of the city's most controversial buildings be saved from demolition.

The NSW government recently rejected a heritage listing bid for the harbourside Sirius building, which has been used for public housing since it was built more than 30 years ago.

Hundreds of protesters marched from Alfred Street, around the Quay, meeting at the base of the Brutalist building on Saturday morning.

The vocal crowd, flanked by police officers, were addressed by several opponents of the building's slated demolition, including Lord Mayor Clover Moore and opposition planning minister Michael Daley.

"If the government applies this policy to other inner city areas, it will destroy the social mix - the very soul of city - and we will fight that all the way," Ms Moore said from the back of a truck in front of the building.

"This housing is needed just as much now, or even more, because the majority of social housing residents in Millers Point have already been dispersed."

The mixed-bag of protesters included unionists, architects and social housing advocates.

The CFMEU granted a Green Ban over the building earlier this week, in an attempt to stall demolition plans.

Michael Daley warned the Baird government any attempt to tear down the building, built in 1979, would be met by fierce opposition.

"We're here to say to Mike Baird, if you try and cheat the people of Sydney out of the Sirius building, when you come down here with your developer and your banker mates, we'll be waiting," Mr Daley said.

Architects Olivia Savio-Matev and Hugo Raggett said the Sirius building held more than just architectural importance in Sydney.

"We're here to support and save the heritage architecture of Sydney, but also to support the residents who are being evicted."

"I think the governments stance on this building is purely a money grab."

The Sirius building is not the only inner-city public housing on the chopping block, with the NSW Government also selling houses in Millers Point to private buyers.