Embattled union boss John Setka has said he will not stand down as CFMMEU Victorian state secretary, plunging the union movement further into crisis.

“There’s no reason for me to resign,” said Setka, who fronted the media Wednesday with his wife, Emma Walters.

The controversial union boss has been facing calls to quit amid claims he had criticised anti-domestic violence advocate Rosie Batty.

On Wednesday, Setka said suggestions he “denigrated” Batty were “fabricated”. But he conceded he told a union meeting that “laws had changed” following Victoria’s royal commission into family violence.

The controversial union figure is also facing court over harassment charges, for which he has said he would plead guilty. Labor and union figures have declined to call for action over the harassment claims, arguing they are still before the courts.

Setka declined to address what he would do if he was convicted of harassing a woman using a carriage service, saying he could not comment. He is expected to plead guilty at a hearing next month.

“I’m elected by CFMEU construction division members, right, every four years,” Setka said.

“They’re the people that I’m beholden to and they’re the ones that pay my wages and I answer to them.

“I don’t answer to anyone else but them. So when an election comes, if they, for whatever reason, see fit to not have me as their secretary, they won’t vote for me.”

It’s understood the ACTU will argue that Setka must lose his position if a conviction is recorded. The ACTU secretary, Sally McManus, has already said that if “any of the allegations are true”, Setka should resign.

But the CFMMEU state secretary is an elected position, meaning Setka cannot be removed unless he stands down or is found to have breached union rules in some way.

Asked if Setka had a problem with respect for women, Walters said the answer was “categorically no”.

She claimed her husband was the victim of a “get-John Setka campaign”, though she added that “both of us have said and done things that we deeply regret and we are trying to heal, together, as a family”.

“I’m simply asking the people who are using our private lives to wage a political war against my husband, John, to stop,” Walters said.

On Tuesday, the Labor leader, Anthony Albanese, said he would move to expel Setka from the party. The Victorian premier, Daniel Andrews, backed the move, and said the government had removed Setka from an advisory board.

Setka was nonplussed about his pending expulsion from the party. “If people want to expel me out of the Labor party over false accusations and over things that I’ve never said, well, then, so be it,” he said.

He admitted telling a union meeting that lawyers had told him that laws had changed following Victoria’s royal commission into family violence. The comments were made when he provided an update to the union national executive about his legal situation, Setka said.

“Some people told me, ‘You shouldn’t have even raised it, John. It’s your own personal thing, it’s nothing to do with them.’” Setka said.

“But I wanted people to know this is where it’s at and what had happened and I gave them a brief outline. Don’t I regret that now?”