Labor leaders have criticised a threat made by CFMEU boss John Setka to "expose" Australian Building and Construction Commission (ABCC) inspectors and make their children "ashamed of who their parents are".

Mr Setka told a trade and construction workers' rally in Melbourne on Tuesday that they were going to expose all ABCC inspectors and make their children "ashamed of who their parents are".

"Let me give a dire warning to them ABCC inspectors, be careful what you do. You're out there to destroy our lives," he said on Tuesday.

"We will lobby their neighbourhoods, we will tell them who lives in that house and what he does for a living, or she, and we will go to their local footy club. We'll go to their local shopping centre. They will not be able to show their faces anywhere."

"Their kids will be ashamed of who their parents are when we expose all these ABCC inspectors."

Speaking in the Upper House in Canberra, Employment Minister Michaelia Cash said the comments had been referred to both federal and Victorian police.

"Thuggish behaviour like this, it doesn't matter where it occurs, should never ever be on display and it should be condemned outright," she said.

"Mr Setka holds an extremely powerful position within the CFMEU and quite frankly after yesterday's behaviour he has shown yet again that he is not a fit and proper person to hold such positions."

Federal Labor leader Bill Shorten dissociated the party from Mr Setka's comments, saying "that's not the way to advance your cause".

Labor frontbencher Anthony Albanese also said the police referral was "appropriate", telling Adelaide radio 5AA the comments were offensive.

"I completely repudiate them," Mr Albanese said.

Thousands of building workers marched through Melbourne's CBD on Tuesday. ( ABC News: Elias Clure )

Victorian Premier Daniel Andrews told reporters it was not appropriate to bring people's families into a "highly charged and emotional debate" about workers' safety.

"I think that if we were having to knock on doors and tell loved ones that their husband or wife or their family member wasn't going to be coming home from work today because there had been an accident, maybe we'd have a different perspective on these things," he said.

"That debate is a fair and proper one to have, but let's not drag people's families into it."

Mr Andrews would not give his view on the ABCC, merely saying the Government had full confidence in Victoria Police to respond to crime.

"That is a matter entirely for the Federal Government. They've passed through their Parliament a set of laws and they can defend or sell or justify or explain to you why those things are necessary," Mr Andrews said.

Thousands of building workers marched through Melbourne's CBD, protesting against the revival of the federal construction watchdog.

Unions argue the ABCC could lead to draconian and dangerous working conditions.

Unions argue the ABCC could lead to draconian working conditions. ( AAP: Joe Castro )

Former ABCC head, now public service commissioner, John Lloyd, said he was concerned by the comments.

He said in the past flyers had been put up at building sites around Melbourne with the name and address of an ABCC inspector.

Mr Lloyd said several similar incidents had been referred to police in the past.

"I am concerned about threats against public servants whether they are ABCC inspectors or other public servants," he said.

"They have a right to go about their work without threats being made against them."

Master Builders Australia chief executive Denita Wawn said she supported notifying police about Mr Setka's comments.

"The threats show utter contempt for our community and its values. Many people will rightly be shocked by this appalling conduct, but this kind of bullying is the building union's stock in trade," she said in a statement.

"It is exactly this kind of behaviour that spurred the [Federal] Parliament to bring back the ABCC and establish the new building code to restore the rule of law to the construction sector.

"Master Builders calls on building unions to publicly commit to allowing government inspectors and regulators to be free from harassment, intimidation and other unlawful behaviour when on building sites and when not at work."

AAP/ABC