Violence has erupted in Melbourne's CBD after rival rallies involving more than 150 anti-racism and anti-immigration activists.

Key points: Riot police and mounted officers overseeing protests

Riot police and mounted officers overseeing protests Anti-immigration group marching to 'celebrate' Australian flag

Anti-immigration group marching to 'celebrate' Australian flag Some protesters seen wearing masks

Anti-racist protesters assaulted a man in the Carlton Gardens following the rallies outside State Parliament on Spring Street.

Three people were arrested and an Australian flag was burned.

More than 100 anti-racism activists had earlier gathered at the steps of Parliament, chanting lines including "No hate, no fear, fascists are not welcome here".

Further down Spring Street, up to 60 anti-immigration activists marched in a rally organisers said was to "celebrate" the Australian flag.

Anti-racist protesters set an Australian flag on fire and clashed with an opposing group. ( ABC News: Sarah Farnsworth )

Scores of police — including officers in riot gear and on horseback — tried to keep the groups separate but some anti-racist protesters managed to splinter off.

Police were able to surround and cordon off the anti-immigration group at the Royal Exhibition Building in Carlton and escort some of the protesters to buses.

Some protesters were seen wearing masks. ( ABC News: Sarah Farnsworth )

The clashes came as Victorian Police Minister Lisa Neville announced new police powers and higher penalties to deal with protesters covering their faces with masks.

"Victorians have had enough of seeing this sort of incitement of hatred and violence," she said.

"It does send a message that if you wear a facemask, you will be removed from a protest. If you wear a facemask and commit an offence, you will go to jail for longer."

Ms Neville said the new laws would be introduced later this year.

Seven people were arrested after clashes between far-left and far-right activists at Coburg, in Melbourne's north, last month.