Punches have been thrown in Melbourne as members of a Reclaim Australia splinter group clashed with anti-racism protesters on the steps of a town hall in the city's inner-east.

Tensions flared on Sunday afternoon as about 70 members of the United Patriots Front (UPF) faced off against about 300 protesters from the group Campaign Against Racism and Fascism at Richmond Town Hall.

"Muslims are welcome, racists are not," the anti-racism protesters chanted.

About 60 police officers and a dozen police horses intervened in the clash, separating the opposing groups.

Police said no arrests were made at the rally, but a man was spoken to in relation to carrying a knife, and is expected to be charged with weapons offences.

Trams along Bridge Road had to be diverted for about two hours, before police moved the rallies on about 2:00pm (AEST).

Local councillor Stephen Jolly said the UPF group was protesting against an anti-racism forum he organised.

Ahead of the rally, the UPF posted on their social media that the rally was to protest the behaviour of the anti-racism group at a previous rally, held in Melbourne's CBD.

"Remember the reason we are fighting against these mongrels!" the group posted.

"On the 4th April these people assaulted an old lady, disrupted what would have been a peaceful rally, they threw horse shit at the cops and let's not forget they burnt the Australian flag!

"They do these things and then have the nerve to say we are against democracy, we're Nazi's, we're are criminals!"