No woman can be free who does not control her own body. That’s been the pro-choice mantra down the ages, no less true now than it ever was, and from it flows the equally fierce conviction that men should keep their noses out of reproductive rights. If it isn’t your womb, your life, on the line here, then what right do you have to interfere in a grown adult’s decision? No wonder that powerful image of the 25 male politicians who collectively approved Alabama’s cruel new ban on terminations beyond six weeks, struck such a chord. The sight of old men desperately clawing back their lost power over women’s lives still triggers a deep, visceral fear. Six weeks! That’s barely a missed period, a time when many women won’t even have realised they’re pregnant.

True, America’s anti-abortion movement has plenty of prominent women in it, and in the UK it’s a female leader of the anti-abortion Democratic Unionist party who (thanks to her sensitive role propping up the British government’s parliamentary majority) represents one of the biggest stumbling blocks to rolling out abortion rights in Northern Ireland. But such nuances get forgotten in the face of men doggedly arguing on Twitter that forcing victims of incest to continue their pregnancies would at least preserve the criminal evidence, or refusing to let their cluelessness about female biology get in the way of a terrible opinion.

Yet the argument that men should all shut up and leave this to women is a risky one, unless we seek a world where virulently anti-abortion men still feel no shame about barging in while pro-choice men hang back for fear of saying the wrong thing. I admire any woman with the courage to say publicly, as the actor Jameela Jamil did this week, that she had an abortion when contraception failed her and “I don’t feel AT ALL ashamed.”

But there’s something uncomfortable about watching women, and only women, feel driven to bare their souls in defence of reproductive and contraceptive rights that have liberated both sexes and which both should be raising hell to defend.

We rarely read or hear about them, but there must be millions of men whose lives were changed for the better by not becoming fathers when they weren’t ready. There will be men who owe their glittering careers and happy families now to the fact that 20 years ago they didn’t have to drop out of university when their student girlfriend got pregnant, or weren’t forced to marry someone they didn’t love. And there will also be men who didn’t have to raise a child in circumstances where they genuinely couldn’t have coped, and whose other children are infinitely better off for it; men who haven’t had to watch their partner struggle through the horror of a pregnancy where everyone knows the child is unlikely to survive, who know how it feels to hold their partner’s hand in the clinic but don’t feel it’s quite their place to talk about it.

If one in three women has terminated a pregnancy, then men with every reason not to take abortion rights for granted, as well as older men who shudder to remember the days before it was legal, must statistically speaking be everywhere; walking down the street, sharing your office, representing you in parliament. The next leader of the Conservative party could very well be Boris Johnson, a man once sacked for trying to hide the fact that his mistress had had an abortion. So who is to say there aren’t men in high public office across America furiously keeping their heads down as the abortion row rages, crossing their fingers that the non-disclosure agreement holds? Yet for every man frantically trying to save his marriage by pressuring his lover to get rid of a baby, there will be couples taking painful decisions together about a much-wanted pregnancy when the tests show something nobody wanted to see.

It’s a hard truth to acknowledge, given it risks creating a dangerous chink in the argument through which anti-abortion campaigners can so easily slide. But for all that abortion should first and foremost be a woman’s choice, it doesn’t only affect women’s lives, and it shouldn’t just be women who are forced to fight do the heavy lifting in fightingfor it. There are, of course, perfectly good reasons for men to stay quiet about their personal experiences. For a man to talk openly about a partner’s abortion can feel pushy and self-centred, and practically speaking risks outing a woman who might not want her reproductive history exposed to all. During the impassioned debate ahead of Ireland’s referendum last year on repealing the ban, many Irish men initially hung back for fear of intruding. Male MPs similarly often defer to female ones in trying to change the law.

But the idea that the only way of galvanising change is by making a public song and dance about yourself may be one of the great curses of modern activism.

Pro-choice men worried about stealing women’s thunder can relax, because they are absolutely welcome to donate to pro-choice charities, sign petitions, lobby their political representatives, and vote for politicians and parties committed to defending reproductive rights. They can stuff envelopes, make tea, go on marches – or look after the kids while their partners do – and cheer from the sidelines to their heart’s content, much as the wives of male activists have done for generations. And where they represent us, they can have the guts to legislate for what they know perfectly well to be the realities of life. No woman can be free who does not control her own body. But we are not the only ones liberated by acknowledging it.

• Gaby Hinsliff is a Guardian columnist

• This article was amended on 20 May 2019. An earlier version contained multiple references to “pro-life”, which is a term the Guardian’s style guide advises against in this context. These references have been replaced with “anti-abortion”.