Rival rallies have been held in the central Victorian city of Bendigo by groups for and against the construction of a local mosque.

Anti-racism groups, operating under the banner of the Bendigo Action Coalition, marched from the city's town hall to Rosalind Park as 400 police watched on.

Hundreds of members of United Patriots Front (UPF) occupied a rotunda in the same park.

Apart from the brief scuffle after a member of the anti-racism group allegedly took a camera from a member of the anti-mosque group, it was a peaceful afternoon.

Victoria Police Superintendent Mick West said four people were arrested during the day — two for carrying knives, one with a flare, and one for minor assault.

"We're very happy with the outcome of today, we've had approximately 600 protesters up in our beautiful city, there's been minimal disruption to our residents," he said.

He said he was not sure whether those arrested were supporting or opposing the mosque.

The UPF rally was the second time in two months that nationalist and far-right groups had demonstrated against the city council's approval of a mosque.

Anti-racism activist Tashara Roberts called on those present at the rally to respect those opposite them, "even the idiots".

"Their views are misguided, they have been shown falsified evidence and photoshopped imagery," she said.

"It is our responsibility as forward-thinking citizens to take every opportunity to educate fellow Bendigonians, and show them why we hold the beliefs that we do, and we must do so peacefully and respectfully.

"We must lead by example, we must show tolerance, even to the idiots."

Following the anti-racism rally, supporters of the UPF chanted "Aussie, Aussie, Aussie, oi, oi, oi", while the UPF's Blair Cottrell criticised the media and politicians.

"[The media] like to belittle our efforts, purely because when you make an enemy weak and ridiculous then people do not flock to that power, and we to the mainstream press, to this weak Government and its corrupt policies are an enemy," he said.

"So they need to lie about our numbers, they need to lie about our strength and spirit, and you can expect those same lies again, but I'd like to see them try this time."

Mr Cottrell told the rally sacrifices needed to be made to protect Australia's identity.

Radical left and right 'using Bendigo': federal MP

The federal Member for Bendigo, Lisa Chesters, said those protesting over the construction of a mosque were using Bendigo as a backdrop for their battles.

Ms Chesters said the regional city was an inclusive one, and the protesters did not represent local views.

"I'm asking people not to judge Bendigo because the people who are running these rallies are not from our town," she said.

"This is a problem across our country.

"It's the radical of the left and the radical of the right using Bendigo as a backdrop for the latest stage for their battles."

But a local activist in favour of the mosque, Tash Joyce, said the idea that those who opposed it were not from Bendigo was incorrect.

"People need to remember that there was a councillor that invited them here and there are people on the anti-mosque group who invited the UPF here," she said.

"They are Bendigo people, we may like to think that this is not a Bendigo response - but it is, they've invited the UPF and made them feel welcome.

"Thankfully, there is a much greater majority of people who want them to piss off."

Superintendent West said he thought "a very small" number of people involved in the rally were locals.

Bendigo City Mayor Peter Cox said he did not understand why the UPF had targeted Bendigo.

He said the council had to better understand what was driving people's fears about Bendigo's future.

"Part of the Australian constitution is that we can practice our belief systems, however there are concerns and fears that people have, we need to talk to those people," he said.

"I really don't understand how people can look at their TV sets in their lounge room about terrorism around the world and link it to building a mosque in Bendigo.

"Some people feel that way, we've got to do some research to help us understand that."

'Major' police operation in Bendigo

Earlier Saturday, Victoria Police Assistant Commissioner Stephen Leane labelled the police presence in Bendigo a "major operation".

There was also a heightened police presence at other mosques across the state, including in Melbourne.

The City of Greater Bendigo ticked off plans to build the mosque in June last year but objectors took the case to the Victorian Civil and Administrative Tribunal (VCAT) to argue the development would cause traffic and social issues.

VCAT dismissed those concerns, but protests have continued, causing tension in the local community, and the city centre was shut down in August when hundreds of people gathered for counter-rallies.

Councillor Cox said the mosque would go ahead despite the ongoing protests.

"Council can't change its mind on the mosque at this point in time, we made our decision 15 months ago, the decision's been upheld by VCAT, the permit's been issued," he said.

"While there's a court of appeal that still has to go they can proceed with their permit at this point in time."

Police were called in to escort Bendigo councillors from their regular meeting last month after anti-mosque protesters began chanting and hurling abuse at them from the public gallery.