Prime Minister Tony Abbott says Muslim leaders who boycotted a meeting with him over counterterrorism laws have acted "foolishly".

At least two groups chose not to attend a meeting with Mr Abbott in Melbourne on Wednesday, including Victoria's peak Islamic body, the Islamic Council of Victoria.

Mr Abbott says the "spirit of the meetings was excellent", despite a small number deciding not to attend.

"Obviously the most important thing is to talk and that's why I was disappointed that a small number of Islamic leaders, community leaders, I thought rather foolishly boycotted that meeting," he told 774 ABC Melbourne.

"I doubt that sort of thing will happen again because its so self-evidently petty to do that kind of thing.

"You can hardly complain that people aren't talking to you, then when they offer you a talk say 'sorry we're not coming'."

He says an Islamic community leader supported his call for all Australians to be "part of Team Australia", a phrase Mr Abbott first used when announcing the new anti-terrorism laws.

The Prime Minister said he believed it was the attitude of the vast majority of Muslim people in Australia.

"One of the leading Imams said to me at the end of one meeting 'we're all part of Team Australia' and 'you're our captain'," he said.

"I don't imagine that from time they wouldn't want a different captain but nevertheless that's what he said."

Race Discrimination Commissioner Tim Soutphommasane said Mr Abbott and the Federal Government should be careful not to cause divisions among the Arab and Muslim community.

Community members expressed concern the term "Team Australia" was divisive.

Mr Soutphommasane said people had the right to demand an explanation from Mr Abbott about what the term meant.

"We're all signed up to the Australian community. We all take a pledge of citizenship if we're migrants and have [been] naturalised," he said.

"And we shouldn't be casting aspersions on a section of the Australian community based on the behaviour of a very small minority."

But Mr Abbott says the term has been widely used, including by Labor politicians.

"I dare say not everyone would use the phrase, I've used it and I'll keep using it, because I expect everyone to be on Team Australia," he said.

Opposition immigration spokesman Tony Burke says the Government has been making a mess of discussions with Muslim community leaders.

"They've handled it in a clumsy fashion, it's in the interests of any Australian that they lift their game and they improve, I hope they do," Mr Burke said.

"There's a good level of goodwill from people wanting to help, the Government needs to be respectful in those relationships and stop making a mess of it the way they have."

'Unjust and hypocritical policy', Muslim leaders say

It came as Muslim leaders in Sydney signed a statement rejecting the proposed counterterrorism laws, labelling them as an "unjust and hypocritical policy".

The signatories included more than 50 Muslim organisations and individuals, including political groups, senior Imams, student and community organisations.

On Wednesday, a meeting with Federal Attorney-General George Brandis and senior members of the Muslim community was postponed.

Space to play or pause, M to mute, left and right arrows to seek, up and down arrows for volume. Listen Duration: 4 minutes 37 seconds 4 m 37 s PM defends 'Team Australia' term Download 8.5 MB

Community spokesperson Rebecca Kay said it was unlikely a meeting will be rescheduled.

"Muslim community leaders that have signed this statement I believe will be boycotting these meetings and boycotting the meetings completely because they completely denounce the new changes to the terror laws," she said.

"They're unfair and they're unjust, so they won't be attending any meetings."

Silma Ihram from the Muslim Women's Association said they wanted genuine conversations.

"The problem that we have is that when we are brought in without knowing what the proposed legislation is, without being consulted genuinely about what solutions we may have or what the problem is in the community that may give rise to any potential radicalisation, then it's not really beneficial," she said.