Immigration Minister Peter Dutton has been overheard quipping about the plight of Pacific Island nations facing rising seas from climate change.

Mr Dutton was involved in a conversation with Prime Minister Tony Abbott, who had just returned from lengthy talks focused on climate change with Pacific Island leaders in Port Moresby in Papua New Guinea.

Noting that today's meeting on Syrian refugees was running a bit late, Mr Dutton remarked that it was running to "Cape York time", to which Mr Abbott replied, "we had a bit of that up in Port Moresby".

Mr Dutton then added, "time doesn't mean anything when you're about to have water lapping at your door".

Social Services Minister Scott Morrison then pointed out to both men that there was a microphone overhead.

There was a large television microphone on a three-metre-long boom pole above the MPs' heads.

The Immigration Minister later refused to answer questions about the incident.

"I had a private conversation with the Prime Minister," he said.

Opposition Leader Bill Shorten criticised Mr Abbott and Mr Dutton over the remarks.

"It was a bad joke by a minister who is a bad joke," Mr Shorten said.

"But the fact that the Prime Minister is laughing along with it reminds me of what Barack Obama said: any leader who doesn't take climate change seriously is not fit to lead."

Greens senator Larissa Waters said Pacific island leaders would not be impressed.

"I think our regional neighbours are going to be utterly horrified at the disdain that our Prime Minister has shown them, and I would want to reassure them that Australians don't have that same sentiment."

Labor Senator Nova Peris said the remarks showed that the Government does not "take North Australia very seriously".

"It makes a mockery of their 'commitment' to North Australia. #capeyorktime #norespect #northernpride," she said in a Facebook post.