When it first happened, several years ago, Greens senator Sarah Hanson-Young couldn’t understand what it was about. She was speaking in the chamber and senators across the aisle started calling out a man’s name.

It was just a name – no allegation – but it hung in the air like a secret.

After it happened a couple more times, it dawned on her. Still, she couldn’t quite believe it. She’d heard the rumours about herself and this man – rumours of an affair she says are false – and she realised to her horror: that’s what they meant.

These male colleagues were threatening her with gossip and sexual smear, right there in the Senate, where anyone could hear, there where she, and they, had the privilege to work as elected representatives.

She told herself, “Don’t look, don’t react, don’t react.” She decided that if she ignored it, it would go away.

It didn’t and now she’s decided she won’t ignore it ever again.

“In the last 18 months or so, it’s gotten worse,” she told The Saturday Paper. “There’s a group that egg each other on. They think it’s funny.”

Last week, Hanson-Young complained about a derogatory comment from Liberal Democrats senator David Leyonhjelm, who told her to “stop shagging men” during a debate on violence against women. When she asked him to apologise, he told her to “fuck off”. The incident has exposed a wider grubby political culture of gossip and sexualised off-mic sniping in the parliament.

While Hanson-Young acknowledges she takes part willingly in robust and sometimes aggressive policy and political debate, she believes that these taunts cross a line.

Hanson-Young says that some senators – mostly men – have called out slurs regularly, including naming men and implying she has slept with them.

She says the taunts are often made too far from Senate president Scott Ryan’s chair for him or anyone deputising for him to hear. Almost always, it happens when the senators’ microphones are off, so the comments are not recorded in the Hansard.

“It happens so much that it is normalised. So much so, that some senators feel able to shout names of men I have supposedly slept with – whom I haven’t by the way, but that’s not the point.”

Leyonhjelm insists he rarely heckles.

“I almost never call out,” he told The Saturday Paper. “It’s almost unknown for me to say anything out loud. There are others who do. She [Hanson-Young] does get heckled.”

Leyonhjelm has acknowledged that he adopts different standards for language being recorded in Hansard and other language he uses in the chamber.

“Outside parliament or when it’s not in Hansard, I am a normal Australian,” he told ABC Radio National.

He told Network Ten later that saying “fuck off” was only unparliamentary if it was captured in Hansard.

Scott Ryan says he expects senators to uphold standards.

“Abuse has no place in the chamber whatsoever,” he told The Saturday Paper. “Whenever drawn to my attention, I remind senators of appropriate conduct and, where appropriate, request a withdrawal. I will continue to do so.”

Hanson-Young complained officially about last week’s comment but Leyonhjelm ignored her request for a withdrawal and apology, instead appearing on Sky News and Melbourne’s Radio 3AW, where he repeated and expanded on his comments. In the radio interview, he named a man with whom he believed she had had a relationship.

Hanson-Young issued a legal notice, requesting that Leyonhjelm apologise and pay compensation. Again, he refused, saying he and his colleagues were tired of her generalisations and her “tendency to collapse in a flood of tears” when challenged.

“She is also wasting her time, money and endless reserves of outrage, because I will not be issuing an apology,” Leyonhjelm said. “Apologising in response to the outrage of a politically correct lynch mob would be insincere. Drawing attention to double standards and misandry is not wrong and does not warrant an apology.”

Hanson-Young says sexist slurs and rumours have been used to undermine women in parliament for too long.

“This is what has happened to me and I know has happened to others,” she tells The Saturday Paper, describing how she has received 800 emails of support – and about 100 condemning her – since the incident was made public.

“It happens so much that it is normalised,” she says. “So much so, that some senators feel able to shout names of men I have supposedly slept with – whom I haven’t by the way, but that’s not the point. Worse still, one of them felt he could go on radio and name such a man. This is bringing what has been previously a dirty Senate chamber taunt into the public realm. But of course, that is the problem with speaking out. Women know that this is the consequence … more shame and more innuendo spread. I know that this is the problem with ‘rumours’ – that while some won’t believe them, others will. It is a horrible dilemma, faced all the time by women and I now know particularly inside this building.”

Her allegation comes as other current and former parliamentarians describe a culture of innuendo and rumour in Parliament House, which is focused frequently on women’s marital status, sexual behaviour and appearance, and is power related. Staff and journalists are subject to it, too.

“I’m not the first and I won’t be the last,” Hanson-Young says. “I don’t believe I would subject to this bullying if I wasn’t a single, divorced woman.”

Hanson-Young says rumours about men may be spread to discredit them as “a good husband or father” but not “to discredit their ability to do their job”.

Former New South Wales premier Kristina Keneally, now a federal Labor senator, says she was relieved that when she entered state politics she was already married with children.

“Women who are not partnered get it much worse,” she says.

Keneally recalls then opposition leader Barry O’Farrell labelling her “Kim Kardashian”, ostensibly because of the matching initials but really to suggest she was vacuous at a time when Kardashian was best known for a TV show, a sex tape and a 72-day marriage.

When it went on unabated, despite several months of protest, she hit back in parliament suggesting the length of O’Farrell’s first marriage – which had been brief – made him “the real Kardashian”.

Keneally was accused of having crossed “an invisible line”, one she believed O’Farrell crossed long before her.

“If you fight back, you’re either going low or you’re playing the gender card,” she says.

Keneally believes the atmosphere is different for women in federal parliament – and not necessarily in a good way. She says this is at least partly because most federal MPs don’t get to go home to their families each night, instead socialising together with “a lot of alcohol” that creates “a very strange, unreal dynamic”.

One senior Labor figure says this kind of atmosphere sometimes leads to women receiving unwanted male attention.

“There are some men who just try it on,” the MP says. “If they get a ‘no’, they shrug their shoulders and move on. If it’s a ‘yes’, happy days for them.”

It all contributes to the rumour mill.

Labor MP for the seat of Adelaide Kate Ellis reveals that when, in her late 20s, she became Australia’s youngest minister as part of the Rudd government, she was forced to confront a tabloid newspaper editor threatening to publish false gossip about her sex life.

Ellis says the newspaper’s correspondent approached her with a rumour that she was sleeping with one of her advisers. He told her he had interviewed people about it and believed it was true, arguing the alleged affair affected Labor’s power dynamic. On that basis, he intended to publish.

When Ellis protested that it was untrue, the journalist said the story would only be pulled if she could persuade his editor it was false.

“I had to personally ring him and convince him that I hadn’t slept with my adviser,” Ellis said.

Ellis told The Saturday Paper it was one of many rumours about her that had done the rounds in Parliament House.

“I’ve had completely unfounded stories that I’m sleeping with [various men],” she says. “That’s the way of bringing you down a notch, but it’s normally done behind your back, not to your face.”

Ellis says it’s different to criticising a person’s capabilities. “It does cross a line because people make up rumours to suggest you’re a slut and then try to suggest it’s a legitimate political story because it somehow goes to your character.”

Ellis says the culture is changing as more women enter politics.

Liberal MP and former minister Sussan Ley concurs: “Each batch of blokes is far more attuned to the world of work that contains talented women.”

But Ley, too, has had her share of rumours. “I’ve been the butt of nasty gossip,” she says. “I don’t know that that’s a female-only thing. Nasty gossip is that you’re sleeping with someone you shouldn’t be. Nasty gossip happens for reasons that are political, not personal. If someone wants to bring you down, it’s never ‘personal’.”

Former senator and Hawke government minister Susan Ryan, who entered parliament in 1975 among the youngest of only six women in the senate, says she was also the subject of speculation about her relationship status because she was divorced.

“No doubt there were remarks,” Susan Ryan says. “… But they weren’t made to my face. I took the path of not showing the slightest bit of concern and, for me, that was the right course. I also felt well supported by male colleagues.”

She acknowledges Hanson-Young feels differently about what has happened to her. She believes the behaviour in the senate has deteriorated.

“I do think there’s a level of toleration of appalling behaviour which seems to be worse than it was a generation ago.”

Kate Ellis and Sussan Ley both say the remark Leyonhjelm made to Hanson-Young is not normal in parliament.

“I think that what’s unusual about the past week is that it has been direct,” Ellis says.

Leyonhjelm’s original comment came during brief speeches on a motion from Fraser Anning, of Katter’s Australian Party, calling to lift import bans on Tasers, pepper spray and mace to protect women, following the rape and murder of Melbourne woman Eurydice Dixon.

The government refused and Labor and the Greens also opposed the proposal, arguing that carrying such things would not stop women being attacked.

The Greens’ Janet Rice protested that the motion put the onus on women and the last thing they needed was “another man in power telling us that we are responsible for violence against us”.

Leyonhjelm says he made the remark because Hanson-Young had said “something along the lines that all men are rapists”. He defends his actions on the grounds that he believes Hanson-Young is a misandrist and that by saying “men” and not “some men” she regularly denigrates men and blames them all for violence against women. Hanson-Young denies this.

“He’s proven that he can’t even remember what the words were,” she says. “I said putting tasers on the streets was not going to make women safer from men.”

Leyonhjelm continues to insist he is standing against misandry, which he says is just as bad as misogyny but given little airing.

Prime Minister Malcolm Turnbull and Opposition Leader Bill Shorten have both condemned his comments and said he should apologise.

Susan Ryan believes parliament should consider what can be done about sexualised off-mic sledging in the chambers. She suggests the presiding officers should convene talks with party whips to establish a way of penalising offenders.

Senate president Scott Ryan told The Saturday Paper: “I have regular discussions with party leaders and representatives, including whips, regarding conduct and operation of the chamber. The issue of language and interjections not part of formal proceedings will be discussed prior to the resumption of parliament in August.”

A week on from their exchange, Leyonhjelm and Hanson-Young are now being accused of milking the issue for political gain.

Not denying he was using the media attention to lift his profile ahead of the next election – and suggesting Hanson-Young was doing the same – Leyonhjelm called the Greens senator “the gift that keeps on giving”.

She responds by asking how people think it feels to answer to allegations – made in parliament – that you sleep with various men and in doing so have the nation discuss whether it is true.

She speaks about seeing a cartoon that appeared in Fairfax Media papers this week, ostensibly criticising Leyonhjelm but also depicting a mirror saying “Sarah is a slut”.

It hit her like a punch in the guts. And now she has decided it’s time to really give it back.