A failure to act swiftly to remove refugees from Nauru and Manus Island could result in "the worst of all scenarios", veteran Liberal MP Russell Broadbent has warned.

"It's one thing to have people die at sea; it's another to have them die in your arms and be responsible for that," Mr Broadbent told Fairfax Media.

"I don't want to be part of a government that takes people to that point."

Mr Broadbent is calling on his leader Malcolm Turnbull to devote all necessary resources of the Prime Minister's office to finding a solution for asylum seekers and refugees on Nauru and Manus Island.