Prime Minister Malcolm Turnbull has weighed into the issue of violent crime in Victoria, saying you would "have to be walking around with your hands over your ears" not to hear the "real concerns" about Sudanese gangs in Melbourne.

But he dismissed a claim that comments from federal politicians had contributed to an increase in the number of reports about racial discrimination.

Earlier this year, Home Affairs Minister Peter Dutton stoked controversy when he said some people in Melbourne were afraid to go out to dinner at night because of gang violence.

"The reality is people [in Melbourne] are scared to go out at restaurants of a night time because they're followed home by these gangs, home invasions, and cars are stolen," Mr Dutton told Sydney radio in January.

The comments prompted a storm of anger on social media and from African community members who accused Mr Dutton of fuelling racial hatred.

Victoria's Equal Opportunity and Human Rights Commissioner Kristen Hilton on Monday said racially-divisive statements about the African-Australian community had contributed to a 34 per cent increase in the number of reports of racial discrimination over the past financial year.

"Earlier this year … there was a lot of reporting around the African gang crisis and we experienced our biggest spike in complaints that we have ever had in relation to race discrimination directly after that period," she said.

Victoria needs to get 'fair dinkum' about crime

In Melbourne for an infrastructure announcement with Opposition Leader Matthew Guy, Mr Turnbull was questioned about the suggestion that there was a link between comments by federal politicians and racism.

"We have zero tolerance for racism," he said.

Malcolm Turnbull said his Government had "zero tolerance" for racism. ( AAP: Julian Smith )

"Australia is the most successful multiculturalism society in the world, it's one of our greatest achievements."

But he said there was anxiety about Sudanese gangs in Melbourne.

"There is certainly concern about street crime in Melbourne. There is real concern about Sudanese gangs," he told 3AW.

"I've heard people, colleagues from Melbourne say there is real anxiety about crime in Melbourne.

"You'd have to be walking around with your hands over your ears in Melbourne not to hear it."

Mr Turnbull took aim at the Victorian Government, saying it was failing to uphold law on the streets.

"You're not going to make it go away by pretending it doesn't exist," he said.

"At some point you've got to be fair dinkum. You've got to acknowledge there is a concern."

PM 'doesn't know what he's talking about'

Victorian Premier Daniel Andrews said he disagreed with Mr Turnbull's comments.

"I would respectfully say the Prime Minister doesn't know what he is talking about," Mr Andrews said.

"We have the best restaurants in the country and they are full, they are absolutely full.

"I don't know that I can offer any further comment than that."

Mr Turnbull said it was "critically important" that people felt safe" walking the streets, and Mr Guy would be tough on crime if he was elected in November.

"If there is a problem frankly I don't really care about someone's ethnicity is," he said.

"We need to make the criminal justice system work for everyone, whether it's gangs who are of Sudanese descent, Ukranian descent or Irish. It doesn't matter."

Pyne quizzed about going out in Melbourne

Federal frontbencher Christopher Pyne appeared to be caught off-guard when asked about the issue during a media conference in Tasmania.

"Are you afraid to go out to restaurants in Melbourne?" a reporter asked.

"No, why? Should I be?" he asked.

The reporter then explained the Prime Minister's comments.

"Oh because of the gangs … the violence. Oh I'm sorry I wasn't following you, I didn't understand the question," he replied.

Mr Pyne said he was not concerned personally because he was not from Melbourne, but it was clearly an issue for Victorians.

"Both senators and members (of Federal Parliament) say that law and order and transport are the number one and two issues in that state, and that Victorians are very concerned about law and order issues in Melbourne in particular," he said.

'No community devoid of crime'

But Graham Ashton, Victoria Police's Chief Commissioner, told ABC Victoria's Statewide Drive that there had been a real reduction in crime.

"We are absolutely responding to the levels of crime that are occurring," Mr Ashton said.

He refused to be drawn on the Prime Minister's comments.

"We've certainly had a deal of African, Sudanese-based offending, but we've also had offending from every other group as well," he said.

"No community's devoid of people in it that might commit crime, and there's good people in every community.

"The Sudanese offending gets a lot of publicity ... but certainly in real terms, we're doing our bit in making sure that we're keeping crime down."