Former prime minister Kevin Rudd has accused John Setka of damaging the Labor Party's brand, arguing his union should have a "long, hard look at itself" as Mr Setka rejects calls to stand down.

Key points: Kevin Rudd says CFMEU leaders have damaged Labor's brand in recent years

Kevin Rudd says CFMEU leaders have damaged Labor's brand in recent years He says the union needs to consider the wider implications of the controversy

He says the union needs to consider the wider implications of the controversy Anthony Albanese is still seeking to expel John Setka from Labor despite union backlash

Labor leader Anthony Albanese moved for the expulsion of the Victorian secretary of the Construction, Forestry, Mining and Energy Union (CFMEU) from the party, after reports he had disparaged the work of anti-violence campaigner Rosie Batty.

Mr Setka denied the claims during a defiant media conference on Wednesday and said there was "no reason" for him to resign from his union role.

But Mr Rudd told Channel Seven the CFMEU should consider the broader implications of the controversy.

"While various elements of the union movement try [to] rally around [Mr] Setka, the bottom line is here the CFMMEU or whatever it's called these days, needs to have a long, hard look at itself in terms of the damage, which certain of its leaders have delivered to the corporate brand of the Labor Party over the years, and Setka's one of them," Mr Rudd said.

"I mean the Labor Party is a party based on fundamentally decent values. And many of us frankly can't stand it when you get people acting out at the extremes."

Another powerful union figure, Christy Cain, defended Mr Setka on Wednesday and said it was Mr Albanese who should consider resigning instead.

Space to play or pause, M to mute, left and right arrows to seek, up and down arrows for volume. Watch Duration: 1 minute 31 seconds 1 m 31 s Maritime Union national president Christy Cain says John Setka did not denigrate Rosie Batty

The national president of the Maritime Union said he was one of 40-plus officials who attended the meeting in which Mr Setka was alleged to have made the comments.

"Mr Albanese's run off false allegations. He has misconstrued what has gone [on] in that room and he should apologise," Mr Cain said.

"Maybe Mr Albanese should resign, I don't know. He should take a good look at himself."

Mr Albanese on Thursday told the ABC he had spoken to several other people in the meeting and stood by his decision to push for Mr Setka's expulsion.

"Unfortunately for Mr Cain, he said in that interview that Mr Setka didn't mention Rosie Batty. Well, that's not what Mr Setka says himself," Mr Albanese said.

"Mr Setka says that his comments about Rosie Batty have been misinterpreted, which is a very different thing from Mr Cain's recollection of the meeting."

Space to play or pause, M to mute, left and right arrows to seek, up and down arrows for volume. Watch Duration: 1 minute 3 seconds 1 m 3 s John Setka decries "false allegations" against him as he vows to stay on at the CFMEU (Photo: AAP/Stefan Postles)

Renewed Government push for union crackdown

Mr Albanese also criticised renewed Government plans for legislation to make it easier to deregister unions and disqualify officials in the wake of the controversy.

The Ensuring Integrity Bill failed to pass the last Parliament but Government minister Angus Taylor confirmed on Wednesday that it would be reintroduced.

"We are absolutely committed to the Ensuring Integrity Bill," Mr Taylor said.

"Labor has opposed it, we will bring it forward again."

Space to play or pause, M to mute, left and right arrows to seek, up and down arrows for volume. Watch Duration: 31 seconds 31 s Anthony Albanese says he'll move for John Setka's expulsion from the ALP

Mr Albanese said unions played a critical role, especially in the construction industry.

"This Government is obsessed with its anti-union position, be very clear," he said.

"It's got nothing to do with John Setka, it's got nothing to do with any individual in the union movement or any particular union."