You might not have noticed that yesterday the Government announced possible changes to the Gender Recognition Act. That’s what ministers wanted: the announcement was carefully made late in the day and was partly obscured by an earlier promise to ban “conversion therapy” that tries to stop gay people being gay.

Why did the Government bury its transgender announcement? The approach was very different last autumn when the Prime Minister herself fronted a prominent media drive which Tory spinners said showed that the Conservatives were inclining towards a system of “self-identified” gender. Yesterday, by contrast, ministers released a deliberately neutral set of consultation questions and kicked decisions on reform into early 2019 by saying the consultation will be open until late October. Conservatives who once rushed to embrace the reforms sought by some transgender lobbying groups are now — behind all their nice words — moving much more cautiously.

So what changed between the autumn of 2017 and this week? There are two factors, one public, one largely private. The public change was brought about by women. Quite a lot of them worry about a system that allows male-born people to take on the legal status of women (transwomen) and thus access spaces and services that the law reserves for women. Especially when some of those transwomen retain male genitalia.

For all that some people suggest it’s somehow prurient or distasteful to talk about penises in this debate, there is, as Nick Robinson put it in some excellent interviews on the Today Programme yesterday, no way to avoid this. The simple fact is that people with penises, whatever word we use to describe those people, are biologically different to people without penises, and that difference matters to many women in a way that cannot be dismissed as bigotry. It is, again, a simple fact that people with penises have the potential to commit certain acts of violence and abuse against others. That fact is the reason Parliament and society accept the concept of single-sex spaces: women have a right to keep someone with a penis out of those spaces.

Upholding that legal right is possibly the founding principle of several women’s groups that have sprung up since the Government first announced its intent to make it easier for people to change their legal gender. Unlike the charities that lobby for transgender rights, the women’s groups — Woman’s Place UK, Fair Play for Women and ManFriday — have no corporate or public sector funding, and not much money at all. They are genuine grassroots political organisations that have sprung up from a concerned public. Those groups have made a difference. Back in the autumn, that point about female-only spaces was either often ignored or dismissed in political debate. Women talking about penises were ridiculed as bigoted cranks, accused of transphobic misinformation. Their meetings were subjected to violent protests (one person has been convicted of assault) and a bomb threat, threats that went shamefully unremarked on by most politicians. Nevertheless, the women persisted: the meetings continued; the campaigns went on; and it made a difference.

Yesterday on the Today programme, Penny Mordaunt, equalities Minister, didn’t dismiss those women as cranks. She said this:

“Those women who are raising those concerns, those are legitimate concerns that we need to address...We will listen to everyone’s voice in this consultation.”

Why has the Government decided to say it will listen to grassroots feminists? That brings me to the less public bit of the story. Some people have been listening to the women’s groups, even if they don’t say so publicly. They include quite a lot of MPs, of all parties. The steady flow of letters and emails from constituents has helped some see that quite a few voters are unhappy about this. (This poll from Pink News underlines that point: 18 per cent of all voters, and 13 per cent of Tories, support allowing people to change their legal gender without medical approval.) That sort of feeling does tell on politicians, even if many aren’t keen to say so publicly, for fear of being accused, like those women’s groups, of nasty transphobia.

(If you doubt the extent of that chilling effect, consider that bomb threat I mentioned. It was made against a Woman’s Place UK meeting in Hastings, in Amber Rudd’s ultra-marginal seat. Even though it would only take a few hundred angry women to switch votes to topple her, Rudd hasn’t yet responded to campaigners’ requests to speak about what the police call a “serious” incident. I find it hard to think of other circumstance in which a former Home Secretary would stay silent about a bomb threat made against a public meeting in their constituency.)

But whether they go public or not, MPs are aware that there are two sides to the gender debate, both equally valid, both worth listening to, and both capable of influencing significant numbers of voters. That message has, slowly, worked its way through the Conservative Party which, never forget, has no Commons majority: it doesn’t take many Tory MPs to cause real trouble for ministers. And in any case, several ministers are extremely well aware of the complexity of the trans debate. Hence the newfound caution in Government circles on trans issues.

This is not just a story about the Tories, however. One of the fascinating things about this issue is that it reveals dividing lines that have almost nothing to do with Right-Left or partisan alignments: there are ardent trans-rights advocates in all parties, and there are politicians and members in all parties who share the concerns of those women's groups.

Labour is, nominally, committed to self-identification of gender, but just like the Tories, the party is in fact deeply divided over the issue and quite a lot of MPs, including prominent frontbenchers, think the issue is more complex (and potentially vote-losing) than the party’s current policies suggests. MPs say a recent meeting of the Parliamentary Labour Party saw many Labour women airing doubts.

In the Labour movement, the dividing line over gender politics sometimes maps onto a split between the new generation of activists in Momentum and the longer-standing groups in, and allied to, the trade unions. Lots of Labour women (and men) sceptical of self-identification of gender have links to the trade union movement which still funds the party, underpins Jeremy Corbyn’s leadership, and exercises real influence in Labour affairs.

In that context, I think a letter in today’s Morning Star is worth reading closely. It describes “systematic attempts to shut down meetings organised by women at which they can discuss potential legislative changes and the impact these may have on any sex-based rights already enshrined in law.”

Such incidents “draw the whole of our progressive movement into disrepute,” the letter says, adding:

“ “Some trans rights activists even continue to justify the use of violence, meaning that many women are simply too frightened to attend meetings that are both public and lawful in order that they may discuss their own rights.

Other women, including ordinary women concerned for their rights, as well as those active within the trade union movement and other political campaigns, are also now anxious and fearful that they will be subjected to such attacks when engaging in any political activity, meetings, or protests.

We are sure that, whatever your view regarding the issues around the Gender Recognition Act, you will agree that it is unacceptable for women to be made scared to engage in political life.”

Now, I’m a dedicated centrist and I happen to agree with every word of that letter. I know paid-up Tories who would too.

But the contents of that letter are not the story here. The story is in the signatories, who include Len McCluskey of the Unite union, as well as several other senior trade unionists. (Lindsey German, a founder of Stop the War and a close friend of Jeremy Corbyn is there too; Andrew Murray, another very senior Corbynista signed a similar letter earlier this year. In short, a very significant and, in Labour terms, powerful group of unionists and activists has raised some quite serious concerns about the violent intimidation of women in the gender debate.

In a previous job as a political reporter, I’d probably have summarised that letter something like this:

“ Transgender activists who use threats of violence to frighten feminist critics are bringing the Labour movement into disrepute, Britain’s top trade union leader has said. Len McCluskey of the Unite union has joined several other close allies of Jeremy Corbyn to warn that “trans rights activists” using threats and intimidation have left many women “too frightened” to engage in political debate.

That strikes me as quite a big deal, and something that others in politics should reflect on. I know that a lot of people in politics (and journalism) veer away from public involvement in this debate, and I think some of the reasons for that are understandable.

But consider these two facts that have come to light this week: the Conservative minister for equalities has said that women have “legitimate concerns” about transgender laws that must be heard and answered; the head of the country’s biggest trade union has said that women who raise those concerns face a “systematic” attempt to scare them into silence.

This isn’t a niche issue, a sideline interest for a few activists and obsessives. This is about how politics works. It’s time for the people who stay quiet to start talking.