The former immigration minister under the Fraser Government has criticised the "ignorant, alarmist voices" in the current immigration debate, singling out Peter Dutton.

Key points: Peter Dutton suggested allowing Lebanese-Muslims into Australia in 1970s was a mistake

Peter Dutton suggested allowing Lebanese-Muslims into Australia in 1970s was a mistake Former immigration minister Ian MacPhee labels discourse, "ignorant, alarmist"

Former immigration minister Ian MacPhee labels discourse, "ignorant, alarmist" MacPhee also singles out One Nation leader Pauline Hanson

Ian Macphee, minister for immigration and ethnic affairs from 1979 to 1982, labelled the comments by the current Immigration Minister "outrageous".

Mr Dutton last week stated former prime minister Malcolm Fraser "did make mistakes in bringing some people in" as part of his immigration policies in the 1970s.

When pressed on those comments during Question Time on Monday, Mr Dutton singled out people of Lebanese-Muslim background.

In a statement provided to the ABC by the Refugee Council, Mr Macphee said the "attack" had prompted justifiable anger from the community.

"We have had a succession of inadequate immigration ministers in recent years but Dutton is setting the standards even lower," he said.

"It [the comments] outraged many community organisations led by former refugees and their children with whom I have remained in contact. Their anger is justified."

Mr Macphee went on to describe Mr Dutton and One Nation leader Senator Pauline Hanson as "ignorant, alarmist voices".

"The Fraser government honoured international law and morality," he said.

"From the Howard government onwards these have been increasingly discarded."

Mr Macphee has been contacted for further comment.

Prime Minister Malcolm Turnbull defended his Immigration Minister amid backlash over his comments.

Mr Turnbull told reporters Mr Dutton was an "outstanding" Immigration Minister, although refused to be drawn directly on the comments in question.

He accused the Opposition of trying to tear down the "thoughtful and committed and passionate" minister.

"They can't stand the fact that he succeeded where they have failed," he said.

The Greens have also criticised Mr Dutton with leader Richard Di Natale calling him a "scumbag" while Senator Sarah Hanson-Young labelled him a "racist bigot".