Immigration Minister Peter Dutton has stood by comments suggesting it was a mistake to resettle many Lebanese-Muslims in Australia, claiming he spoke the truth and would not be intimidated.

Key points: Dutton says he was having an "honest discussion" about immigration

Dutton says he was having an "honest discussion" about immigration Shorten has described the comments as disgraceful and called on him to "immediately apologise"

Shorten has described the comments as disgraceful and called on him to "immediately apologise" Labor MP Anne Aly links comments to death threats made to her family

Mr Dutton singled out the community during a debate on Immigration in Parliament earlier this week, saying many people charged with terror-related offences came from that background.

The comments have hurt and angered Lebanese-Australian Muslims, but the Immigration Minister said he was having an "honest discussion" about immigration.

"The vast majority of Lebanese-Australians are law-abiding, hard-working, good, decent people who are besmirched by a small element within their community who are doing the wrong thing," he said.

"I have been factual in what I said and I want to make sure that we have the best possible country."

Labor has accused Mr Dutton of undermining the ability of security services to work with the Lebanese community on counter-terrorism measures — a claim Attorney-General George Brandis rejects.

Mr Dutton has now criticised Mr Shorten in return.

"Bill Shorten can carry on being part of the tricky elite in this country," Mr Dutton said.

"He can talk double-code to people, he can be tricky in his language. I'm not going to be intimidated by it."

Mr Shorten described Mr Dutton's comments as disgraceful and called on him to "immediately apologise".

Comments 'uninformed and unacceptable': Lebanese Muslim Association

Coalition MP Andrew Laming defended Mr Dutton's comments on Wednesday, saying he did no more than speak the "prima facie truth" about ethnic communities.

But Lebanese Muslim Association president Samier Dandan said Mr Dutton's comments were uninformed and unacceptable.

"Let us not beat around the bush here, what Mr Dutton said was racist, what he implied was racist, and the lack of outrage in Parliament reflects on the racism underscoring much of how we talk about minorities in Australia," he said.

"The Australian Lebanese community is not political fodder."

Mr Dutton also criticised former prime minister Malcolm Fraser, saying he "did make mistakes in bringing some people in" as part of his immigration policies in the 1970s.

Labor MP Linda Burney said the Immigration Minister had "sunk into a major sulk" after facing criticism for his comments.

"In the seat of Barton there are many Lebanese-Australians who have proudly contributed to the economy… they are friends of mine and are remarkably decent people," she said.

Ms Burney said the community was "extremely upset" and Mr Dutton had caused enormous distress.

"These are people who have been in this country all of their lives," she said. "They deserve our respect".

Dutton condemns threats

Labor MP Anne Aly, the first Muslim woman to be elected to Parliament and an expert in counter-terrorism, said the Immigration Minister's comments were "extremely disappointing" and feared they were made in malice.

Dr Aly linked Mr Dutton's comments to death threats she and her family received along with an email telling her to go home and "take all her terrorist friends with her".

Mr Dutton condemned the threats, but said Dr Aly should be asking questions of her party leader.

"I would condemn anybody that has made any death threat, of course I would," he said.

"But again, the question she should be asking Bill Shorten is why did he seek to whip this up into an issue of political advantage for himself?"