Peter Dutton has hit back at Federal Greens MP Adam Bandt who labelled the Immigration Minister a "terrorist" at a rally in Melbourne protesting against the treatment of men who remain on Manus Island.

More than 1,000 people gathered at the State Library before staging a mass sit-in at Federation Square to show their support for 600 refugees and asylum seekers who remain at the Manus Island detention centre in Papua New Guinea, despite its official closure on Tuesday.

Trams and traffic were brought to a standstill, while access to the train station was disrupted. Hundreds also gathered at a similar rally in Sydney.

Sorry, this video has expired Peter Dutton brushes off being labelled a "terrorist"

The men on Manus Island say they do not feel safe in the replacement accommodation provided for them in the community.

Mr Bandt told the crowd the men had been "thrown in prison" by the Australian Government.

Many protesters focused on Immigration Minister Peter Dutton. ( AAP: Mal Faircough )

"These people have committed no crime other than to do what every single one of us would do if we thought our lives, or our family's lives, were at risk," he said.

Mr Bandt likened the Immigration Minister to a "terrorist" for "threatening people's lives".

Protesters gathered outside Flinders St after speeches at the State Library. ( Twitter: Michelle Bennett )

"If the definition of terror is to use violence and threaten people's lives for political purposes, then Peter Dutton is a terrorist," he said.

"To look at the face of Peter Dutton is to stare into the eyes of someone who is prepared to kill people for political gain, and it's time he was held to account for this crime against humanity."

He also criticised Labor for reopening the offshore processing centre while in government, and urged them to join the Greens in their push to bring the men to Australia.

The rally began at the library before moving through the city. ( ABC News: James Hancock )

Mr Dutton said the Government stood by its record on preventing asylum seekers dying at sea, and dismissed the "terrorist" remark.

"I think that says more about Mr Bandt and the Greens than it does about me. The Australian Government has stopped deaths at sea. We've got every child out of detention," he said.

"Mr Bandt, when he was in Government with [former Prime Minister] Julia Gillard, presided over 50,000 people coming on 800 boats and 1,200 people drowned at sea.

"I've not had a single person drown at sea on my watch."

The minister told the ABC in Perth it was "disappointing" that the Greens "seem to be spoiling for some sort of fight" over the Manus Island issue.

Up to 20 protesters waited at the entrance of Crown Casino in Perth, where Mr Dutton addressed the Asia-Pacific Regional Conference between Germany and Australia.

Demonstrators in Perth protested outside the venue where the Immigration Minister was to give a speech. ( ABC News: Eliza Laschon )

'The Government's position is absolutely clear'

The UN on Friday said the Federal Government should provide immediate protection, food, water and other basic services to men, calling in an "unfolding humanitarian crisis".

The UN High Commissioner for Human Rights said there were serious concerns about the safety and wellbeing of the men, and both Australia and PNG were responsible under international human rights law to protect them.

The protesters called for the men to be brought to Australia. ( ABC News: James Hancock )

Loading

In response to the UN's statement, Mr Dutton's said there would be no change in the Government's policy and asked "that people be fully informed of the facts in relation to these matters".

"The Government's position is absolutely clear, it's meant we've been able to stop boats and we've been able to stop deaths at sea," he told reporters in Perth.

"If we allow people who've sought to come to our country by boat, to come to Australia to reside permanently then that will mean that people smugglers are back in business."

Opposition Leader Bill Shorten, who has urged the Federal Government to consider New Zealand's offer to help resettle 150 of the men, said the UN "had a point".

"We don't want to see the people smugglers back in business, but I think there is something going on at Manus which is deeply disturbing to the Australian people," he said.

"Where you have got 600 people without food and water for days, the Government needs to take an active interest in their welfare."

Meanwhile, Papua New Guinean soldiers have been instructed not to allow anyone to bring food to the refugees inside the detention centre.

Food, power and water supplies to the centre were cut when the site was officially closed.

A soldier at the gate to the Lombrum naval base, in which the centre is located, says a church group with food for the refugees was turned away on Thursday.

On Friday, soldiers prevented a locals from landing a boat carrying food near the centre.

The 600 men inside say conditions are worsening but they remain determined to stay.