Victorian construction union boss John Setka has defied calls from both sides of politics to step down, after days of intense scrutiny over his position.

Key points: Mr Setka has said he will remain secretary as long as union members want him there

Mr Setka has said he will remain secretary as long as union members want him there He called reports he made disparaging remarks about Rosie Batty "completely false"

He called reports he made disparaging remarks about Rosie Batty "completely false" Labor leader Anthony Albanese has moved to expel the union leader from the party

Labor Party officials yesterday suspended the party membership of Mr Setka, who is the Victorian and Tasmanian secretary of the Construction, Forestry, Mining and Energy Union (CFMEU).

Federal Labor leader Anthony Albanese moved to expel Mr Setka from the Labor Party after reports he disparaged the work of anti-violence campaigner Rosie Batty.

The report in The Age newspaper said he had told a union meeting that men had fewer rights because of Ms Batty's work.

"What was reported in relation to Rosie Batty was completely false," Mr Setka said at a defiant press conference in Melbourne today.

"There's no reason for me to resign," Mr Setka said.

"I'm staying as the secretary."

Mr Setka was supported at the press conference by his wife Emma Walters. ( AAP: Stefan Postles )

He said he did not denigrate Rosie Batty during his speech, but instead spoke about changes which had been made since a royal commission into family violence.

The Victorian branch of the CFMEU elects its leaders through its members, who Mr Setka called his "bosses".

"As long as I am elected leader, one thing won't change: I will always put our members first," he said.

Calls to quit blamed on 'dirty politics'

Pressure has been mounting on Mr Setka to stand down, with Labor politicians joining Prime Minister Scott Morrison in calling for his resignation.

Australian Council of Trade Unions (ACTU) secretary Sally McManus, who has been under pressure to respond to the issue, released a short statement after Mr Setka's press conference.

She said she had heard what Mr Setka had to say and would meet with him on Thursday.

"I have arrived back in the country and have consulted with union leaders who are concerned by Mr Setka's words and actions which are not compatible with our values and have impacted on our movement," Ms McManus said.

"I have heard what Mr Setka had to say today. I have sought a meeting with him tomorrow to discuss these matters. I will have more to say following this meeting."

Mr Setka is a heavyweight in the CFMEU, which has long-standing links to the Labor Party and provides political donations on a state and federal level.

He said there appeared to be "some sort of push" inside his union to remove him.

Space to play or pause, M to mute, left and right arrows to seek, up and down arrows for volume. Watch Duration: 31 seconds 31 s Anthony Albanese says he'll move for John Setka's expulsion from the ALP

"I've got a view of where the union should be in regards to the ALP, and some people probably don't share that view," he said.

"So there's a lot of dirty politics at work here."

Mr Setka was flanked by his wife Emma Walters at the press conference, who said her family had been "dragged through the mud".

"We've both said and done things that we aren't proud of, but this is not an opportunity to get John Setka," Ms Walters said.

Electrical, maritime unions back Setka

Following the press conference, Mr Setka was backed by Electrical Trades Union Victoria secretary Troy Gray, who called for Mr Albanese to end his bid to have Mr Setka removed from the party.

"Yesterday Anthony Albanese made disparaging remarks about Mr Setka and his alleged 'views'," Mr Gray said.

"Today we've learned they were based off of a complete fabrication and Albanese needs to withdraw."

Mr Setka was also backed by the national president of the Maritime Union of Australia (MUA), Chris Cain, who was at the meeting in which Mr Setka was accused of making disparaging remarks about Ms Batty.

Mr Cain said "false allegations" and "misinformation" were being peddled and called on the Opposition Leader to apologise.

The MUA is a member of the national CFMMEU, following a merger of the two unions.

Yesterday, Mr Albanese said the move to expel Mr Setka was due to a range of factors, including the reported comments about Ms Batty.

He also pointed to comments Mr Setka made about inspectors working for the Australian Building and Construction Commission, which drew widespread condemnation at the time.

"If people want to expel me from the Labor Party over false accusations and over things I've never said, then so be it," Mr Setka said.

Earlier this year, Mr Setka was charged with using a carriage service to harass a woman.

Mr Albanese yesterday said the moves to remove him from the ALP were unrelated to those court proceedings.

Victorian Premier Daniel Andrews said he would not weigh in on whether the CFMEU should dump Mr Setka, but confirmed he had removed the union leader from a government advisory board.