Anti-Islam protesters and anti-racism activists have clashed in Melbourne despite police attempts to keep the opposing groups apart.

Reclaim Australia splinter group United Patriots Front (UPF) planned to rally in front of the Richmond Town Hall on Sunday afternoon.

However, more than 100 protesters from a counter-rally by Campaign Against Racism and Fascism stationed themselves near the Town Hall before they got there.

A huge police presence, including mounted police, were at Sunday's rally and kept the two crowds apart.

The two groups traded verbal insults and some punches were thrown when they confronted each other near the town hall. Police scuffled with some protesters as they tried to keep them apart.

The rally started with about 50 UPF members marching to the town hall chanting "Aussie, Aussie, Aussie" and "I am Australian", and holding the Australian flag.

But it was not long before the much larger crowd from Campaign Against Racism and Fascism drowned out their chants with slogans such as: "f*** off Nazis" and "Muslims are welcome, racists are not".

The rally started about 1.30pm and stopped traffic around the town hall for more than 30 minutes.

UPF said the gathering was a response to the left-wing violence members of its group had experienced during clashes at Federation Square in April.

UPF rally organiser Shermon Burgess told supporters that they were protesting against radical Islam and denied he was racist.

"I cannot even have my point of view without being labelled a bigot or a racist," he said.

Police had to separate angry crowds on April 4 after tensions flared and scuffles broke out over issues of extremism and violence.

Victoria Police said they were at the rallies to maintain public safety.

"Victoria Police respects the right of the community to express their views peacefully and lawfully, but will not tolerate those who break the law," a statement said.

A police spokeswoman later said while there were no arrests at the protest, a man will be charged on summons after he was allegedly carrying a knife.

As Sunday's protest ends, another is being planned.

Uniting Against Islamophobia spokesman Stephen Jolly said the group will show their support at a rally on July 18 - the same day and location as Sunday's event staged by Reclaim Australia.