A Sydney lawyer says she is prepared to fight in the High Court over a decision by ASIO to cancel the passports of 20 Muslim men.

The SMH reported this morning that ASIO had cancelled the passports over fears the men, who are from Sydney, would travel overseas to engage in "politically motivated violence".

The move comes after the AFP last week charged two men with allegedly attempting to join the conflict in Syria.

At least four Australians are known to have been killed in the Syria conflict, and intelligence agencies are thought to be tracking about 100 Australians active in the fighting.

Lawyer Zali Burrows says she has been approached by 15 of the men whose passports have been cancelled.

Ms Burrows says the men have received letters requiring them to surrender their passports and they have been told they are an adverse security risk.

She says the men are worried and she is not aware of any of them wanting join the fighting in Syria.

"Basically they're suspected of being terrorists or wanting to go to overseas and participate in jihad war, or whatever other reasons," she said.

"Some of these guys were just young guys wanting to go on holiday to Bali, they even packed their board shorts. Others wanted to go and visit a sick relative."

One of the men, 19-year-old Abu Bakr, told Fairfax he had been targeted for speaking out about violence against Muslims.

"It is a 10-page letter saying I had a jihadi mentality ... I have never been approached by ASIO to talk about this," he said.

"We have been treated unjustly. My record is clean, shiny gold. I am not a criminal."

ASIO says it can cancel passports on security grounds and that withholding passports is an important means of preventing Australians from acting against national security.