The controversial Sirius site in The Rocks could house a sex shop under planning changes proposed by the NSW government, which also envisage the demolition of the Brutalist-designed public housing block.

A rezoning of the site would permit a number of uses including a hotel, child care centre, registered club and "restricted premises", which include sex shops and similar premises, as part of the NSW government's plan to sell the building to fund new social housing.

Architect Shaun Carter, the chairman of the Save Our Sirius Foundation, said "anything is possible if it is permissible under the zoning".

"Sydney is desperate for affordable housing, Gladys [Berejiklian]'s number one priority in her premiership, then why is she demolishing perfectly good and robust housing that could make the city run more efficiently?" he said. "Seems odd, doesn't it?"