Opposition Leader Bill Shorten says Peter Dutton's apology for joking about the impact of climate change on Pacific island nations is not about what he said, but the fact that he was caught.

The immigration minister was picked up by an overhead microphone making a quip about "water lapping at your door" at a media opportunity with Prime Minister Tony Abbott in Canberra on Friday.

The comment has been heavily criticised by Pacific island leaders.

"I made a mistake, I apologise to anyone who has taken offence. It was a light-hearted discussion with the PM and I did not mean any offence to anyone," Mr Dutton told Sky News on Sunday.

But Mr Shorten said if Mr Dutton was genuinely sorry, he would have said so two days ago.

"It was offensive and demonstrated just how arrogant and out-of-touch this government is," Mr Shorten told AAP.

He also questioned why it was Mr Dutton apologising and not Mr Abbott.

"The prime minister was guffawing at the plight of Pacific Islanders and he thinks that's acceptable? Is this what Tony Abbott and his ministers laugh about in their private conversations?" Mr Shorten said.

Deputy Labor leader Tanya Plibersek said the minister's apology didn't go far enough.

Mr Dutton had managed to insult millions of people in one go and he should have apologised to Pacific leaders and indigenous people in Cape York, she told ABC TV on Sunday.

"The problem is that they (the government) don't care that these Pacific Island nations are facing an existential threat," she said.

Noting that a meeting to discuss the resettlement of refugees was running a bit late on Friday, Mr Dutton had quipped that it was running to "Cape York time", to which Prime Minister Tony Abbott replied: "We had a bit of that up in Port Moresby."

Mr Dutton then said: "Time doesn't mean anything when you're about to have water lapping at your door."

Papua New Guinea Prime Minister Peter O'Neill described Mr Dutton's comments as "most unfortunate".

"People are being forced off the land where their families have lived for thousands of years," he said.

"Connection to the land is very important for Pacific people so having to leave their land is heartbreaking for many people."