Maritime workers have expressed anger at the removal of an "iconic" fig tree from the grounds of Admiralty House in Sydney, as the Governor-General's office insists it had to go because it was threatening the site's historic Marine Barracks.

Key points: The Moreton Bay fig appeared healthy and stood for an estimated 100 years

The Moreton Bay fig appeared healthy and stood for an estimated 100 years The Governor-General's office said its weight was damaging the historic Marine Barracks on the site

The Governor-General's office said its weight was damaging the historic Marine Barracks on the site It will be replaced with a six-metre Norfolk Pine

The Moreton Bay fig was a prominent feature on the horizon at the harbourside residence of the Governor-General, Sir Peter Cosgrove, with estimates it had stood for a century.

Yesterday there was shock among maritime workers as its branches were cut down over five hours, leaving only the trunk behind.

Worker Simon Johnson said the tree looked "as healthy as anything".

"The foliage in summer was absolutely gorgeous and thick," he said.

"It's an iconic Moreton Bay fig on the end of Kirribilli Point. It's a well-loved landmark in Sydney."

Mr Johnson said colleagues started ringing around to try to find out what was happening and stop the removal.

Locals have questioned why the fig was removed. ( Supplied: Simon Johnson )

"It's too late now, it's been hacked to death unfortunately," he said.

He also questioned whether the tree was removed under the distraction of Melbourne Cup day so fewer people would notice.

"It's absolutely tragic and the more I think about it the angrier I am," he said.

North Sydney Mayor Jilly Gibson said usually council would need to be notified of tree removals, but this was not the case on the grounds of Admiralty House.

She estimated the tree was 100 years old and said she'd fielded many calls from concerned residents wanting to know why the "extraordinarily healthy" fig was gone.

A wall inside the barracks was badly damaged under the weight of the tree. ( Supplied: Governor-General's office )

The Governor-General's office said the tree self-seeded on a steep bank immediately behind the Marine barracks and the weight was pushing against the rear wall.

It had already irreparably damaged a retaining wall.

"Engineers have reviewed the situation and say the retaining wall could fail at any time," a spokesman said.

"If this was to occur the historic barracks would be severely damaged if not destroyed."

Heritage experts advised the fig should be replaced with a mature Norfolk Pine.

Roots from a decades-old fig tree at Admiralty House were intruding into the Marine Barracks. ( Supplied: Governor-General's office )

The replacement, which will be six metres tall, will be planted at the top of the bank.

"These works will protect the Marine Barracks, ensuring an important part of Sydney and Australia's history remains intact, and will preserve the heritage value of the Kirribilli Point Battery precinct," the spokesman said.

Cr Gibson said it was reassuring there was a legitimate reason for the fig's removal.

"They had to make a terrible choice between the tree and this very precious heritage structure. The structure won out," she said.

"Mystery solved."