The Greens and Labor will support a no-confidence motion in Home Affairs Minister Peter Dutton over the visas for au pairs affair.

Pressure is mounting on the failed Liberal leadership challenger over his intervention in two visa cases to save to European nannies from deportation in 2015.

Greens MP Adam Bandt will move the motion when federal parliament resumes next week, with fellow crossbencher Andrew Wilkie and Labor set to support him.

READ MORE Peter Dutton says he has records of 'quirky' visa requests from Labor MPs

The vote is set to be tight with the government's numbers in the lower house diminished by former prime minister Malcolm Turnbull's resignation and Kevin Hogan's move to the cross bench.

"Peter Dutton has failed to explain why he misled parliament and now he must go," Mr Bandt said.

The cases relate to an Italian au pair who was linked to Mr Dutton's former Queensland Police colleague, and a French woman who had worked for a relative of AFL boss Gillon McLachlan.

Mr Bandt says Mr Dutton misled parliament by saying he had no personal connection to the cases.

"Peter Dutton has been caught out, he has no explanation and if he won't resign, parliament should make it clear it has no confidence in him," the Greens MP said.

But Mr Dutton has strenuously denied any misconduct, saying he's kept a list of Labor MPs who have come to him with "quirky" visa cases.

"Labor can ask me 10 questions every day when we go back if that's what they want to do, but they'll get a whack back," the minister told reporters in Brisbane.

"To say I had some personal link or that I was acting on behalf of, you know, somebody that I was personally associated with is complete nonsense."

Labor leader Bill Shorten said the home affairs minister was under the pump because of internal pressure relating to his role in dumping Mr Turnbull.

"It's not what you know it's who you know under this Liberal government. That's not the way to run an immigration policy," Mr Shorten told reporters in Brisbane.