Leading figures in the architecture and art communities have condemned the New South Wales Government's decision to deny the Sirius building heritage listing, calling its potential demolition "a cultural tragedy".

On Sunday, Environment and Heritage Minister Mark Speakman announced the public housing building which overlooks Sydney Harbour would not be heritage listed and would be replaced by a new development.

While the building has long polarised the public for its visual aesthetic in the prime harbour location of The Rocks, many believe its demolition will be a great loss to Sydney's architecture and history.

"The demolition of the Sirius building would be a cultural tragedy," artist and former Archibald winner Del Kathryn Barton told 702 ABC Sydney.

The Government says the Sirius building will not be replaced by a "glass skyscraper". ( ABC News: Andrew Griffits )

"Around the world people are celebrating brutalist architecture ... I'd like to think that Australia culture can have that cultural sophistication.

"It's a really unique and important monument that deserves to be preserved."

The Government's decision was made despite recommendations from the Heritage Council in February which unanimously recommended Sirius be protected.

Mr Speakman said the financial implications of heritage listing the building "greatly outweighed the opportunity to get another $70 million or so to reinvest in social housing".

"[Sirius] would need another $15 million spent on it to bring it up to the building code; it's not fit for habitation at the moment," he said.

"You could get a lot more public housing by not listing it, by replacing it with something that is at an appropriate height and scale and look for The Rocks and getting an extra 240 or so among the neediest in our community into public housing."

Mr Speakman said the Sirius building would not be replaced by a "glass skyscraper" but that Property Development NSW would submit a proposal that "respects the feel of The Rocks ... will be shorter in the middle, a bit higher on the sides so there will be an unobstructed view from the Bradfield Highway".

The Sirius building was built in 1979. ( Supplied: Monica Kovacic )

However Shaun Carter, the NSW president of the Australian Institute of Architects and chairman of the Save Our Sirius Foundation, said the decision should be more "than just a beauty contest".

"It has social significance, it has cultural significance, it has environmental significance," Mr Carter said.

"This Government has a terrible record of affordable housing but it keeps on ... selling public buildings so it can get more revenue so it can make decisions against the budget.

"It can appropriate the funds elsewhere to build social housing."

Mr Speakman rejected Mr Carter's comments, saying they were: "All care, no responsibility."

Preserving Sydney's brutalist buildings

Sydney's brutalist architecture of the 1960s and '70s was culturally significant and the style was being re-examined and preserved around the world, according to Ms Barton.

She said in countries like Japan, concrete structures were just as "celebrated as a surface as a garden is" rather than considered a degenerating feature as many in Sydney do.

2013 Archibald winner Del Kathryn Barton says the brutalist style of the Sirius building is unique. ( ABC News: Sally Block )

"I'm very rarely moved to reach out, but I do take very personal pleasure in the building and it has unique and important beauty," Ms Barton said.

"World-class cities need aesthetic architecture.

"I would love to shoot a film there ... It's an incredibly unique and intriguing space."

The Sirius building was built in 1979 for public housing tenants and has not be upgraded since.

Tenants were moved out of the building last year.

Mr Carter said while cities had to grow and change, structures which contributed to the "fabric" of the city should be preserved.

"Out of that need for social housing, Sirius was born," he said.

"Sirius not only has this architectural exemplary value but it also has social and cultural value — the history and the stories of the people of Sydney and how Sydney grew and changed."