The ACTU secretary, Sally McManus, has demanded John Setka apologise for comments he is reported to have made made criticising anti-domestic violence campaigner Rosie Batty, adding pressure on the high-profile union official to resign.

The leader of the union movement directly addressed the controversy surrounding Setka for the first time on Monday afternoon, as the ACTU and Labor faced pressure over a response one anti-domestic violence campaigner labelled “pathetic”.

Setka, the Victorian secretary of the CFMMEU, has claimed he was “taken out of context” following an Age story that reported he told a union meeting Batty’s advocacy had led to men having fewer rights. He told the paper he had not criticised Batty and had great respect for her. Setka could not be reached for comment.

Last month, Setka also told a Victorian court he would plead guilty to charges of harassing a woman.

On Sunday, the ACTU condemned “all acts of family and domestic violence” in a three-line statement that did not refer to Setka’s criticism of Batty and noted the harassment charge was before the courts.

On Monday afternoon, McManus went further, while also praising Batty for making an “incredible contribution to efforts to end family and domestic violence”.

“If the comments attributed to John Setka reported over the weekend are correct, they are appalling and totally unacceptable,” she said.

“They do not reflected out values or the work that unions do to campaign against family and domestic violence and should be immediately withdrawn and apologised for.”

Batty, whose son Luke was killed by her estranged husband in 2014, told the ABC on Monday that Setka’s remarks were “ludicrous” and an example of “somebody whose leadership role really needs to be challenged”.

“I’m quite astonished with those comments,” said Batty, who was on Monday appointed an Officer of the Order of Australia for her advocacy.

“Rather than feel, you know, men have less rights, we have so much more to gain by recognising that moving forward we’re [moving to] a place of equality.”

The Victorian premier, Daniel Andrews, had been criticised for not personally commenting, though he has since demanded Setka apologise.

The federal Labor leader, Anthony Albanese, described Setka’s remarks as “totally unacceptable if that’s what he said”, but also played down his significance within the party.

Speaking before McManus’ fresh comments, Phil Cleary, an anti-domestic violence campaigner who has in the past held roles with the Electrical Trades Union, lashed the response from Labor and the union movement as “pathetic”.

“The statement from the ACTU is not good enough,” Cleary told Guardian Australia. “It’s not good enough talking in generalities about condemning violence against women, everyone will do that.

“The question is, what about when a key person in your movement says that a campaigner, a woman who lost her son to a violent man, is the reason for men losing their rights?”

Cleary, whose sister was murdered by her former partner in 1987, said he had great respect for Andrews for establishing a royal commission into family violence. Andrews has won plaudits for speaking out against men’s violence, and often uses social media to call out misogyny.

Tim Lyons, a former assistant secretary of the ACTU, has said in a tweet that Setka “has to go”, while Jake Wishart, president of Victorian Trades Hall’s Young Workers Centre, also said Setka “should step down immediately”.

Congratulations to Rosie Batty on being honoured with an AO. Rosie is my hero, she is one of the bravest women I know, her work has profoundly changed Australia for the better. If some men feel their power is threatened by her then I say it’s about time https://t.co/EXXTsteUpU — Jenny Mikakos MP (@JennyMikakos) June 10, 2019

Setka’s role as state secretary is elected, meaning it would be hard to remove him unless he stood down. He is a member of the Labor party.

Sources close to the ACTU said that internal discussions around Setka’s future began some time ago and are ongoing. They said the ACTU would argue it would not be appropriate for Setka to remain in his role if a harassment conviction was recorded.

The CFMMEU was contacted by Guardian Australia for comment.

Asked to respond to Cleary’s criticisms, a Victorian government spokeswoman pointed Guardian Australia to a statement from the premier.

“These comments are unacceptable and he should apologise for them,” Andrews said in the statement.

He described Batty as a “great Australian” and said the government was “proud of her work in combatting family violence”.

“Given that a number of these matters remain before the courts, it would be inappropriate to comment further,” Andrews said.

Albanese said on Sunday he was “on Rosie Batty’s side” and described her as a “great Australian”.

The Labor leader said he had never met Setka, who he said did not have “any great significance” within Labor.

On Sunday, the ACTU said it would “continue to advocate for workers’ rights to a safe home, community and workplace and take a stand against family and domestic violence”.