A heavy police presence is visible ahead of an anti-Islam protesters rally in Melbourne.

Cowards with covered faces who went to a Melbourne rally looking to start a fight won't be allowed to get away with their criminal behaviour, police say.

Seven people were arrested as more than 500 people rallied in heavy rain against both Islam and racism in Coburg on Saturday.

Victoria Police say people who hid their faces are to blame for violent clashes between rival groups during the rally at Bridges Reserve, which began about 11am.

Five people were arrested for riotous behaviour as well as assaulting and hindering police, while two more were arrested for carrying knives and other weapons.

Some draped in the Australian flag and most with their faces covered were drenched in pepper spray and hit with Australian flag poles as hundreds of police tried to separate the groups numerous times during the rally.

Some said they were part of the anti-Islamic group UPF and the True Blue Crew.

Others were marching as part of the No Racism in Moreland rally.

When they were separated the two groups did their best to find other ways to fight each other.

They ran from one side of the park to another, or across the street into the grounds of a local primary school in an attempt to break police lines that blocked entrances between the groups.

"There was riotous behaviour and it was appalling ... the community shouldn't have to put up with it," Commander Sharon Cowden told reporters.

She said police will set up a task force to investigate anyone who hid their identity while committing a crime during Saturday's rally.

"What happened today is not on. We will be looking at the footage, finding out what else we can do, to track these people down and bring them to justice," she said.

UPF's Blair Cottrell described the rally as a "great success" but he was interrupted by a masked follower when asked if the fighting between the two groups weakened their message.

That man yelled abuse at reporters covering the rally, swearing at them and accusing the media of twisting his words.

The anti-racism protesters, many of whom also had their faces covered, threw punches and oranges over police lines as well as calling their rivals "Nazi scum".

Ms Cowden said police had been prepared for the possibility of violence on Saturday, which she blamed on a small group.

"If you look at the footage you'll see police step in, quell the violence, and then what happens is you get splinter groups - they break away and they look for a fight somewhere else.

"You get the extreme ends and people coming along and all they're looking for is a fight."

It's not the first time the groups have clashed, and police have previously separated them during other rallies in Bendigo and Melbourne.