The way I remember it, we sat for at least 10 minutes in silence. No one wanted to be the first to speak, for fear that would look too domineering. Even the people who'd organised the group stayed quiet, each of them reluctant to play the role of "leader", with its dread, patriarchal associations. So we stayed seated in our circle of wooden chairs, earnestly mute. When one man did finally start talking, it was in a whispered mumble, lest he be deemed excessively assertive.

Ah, happy memories of student life, specifically my first (and, I fear, last) meeting of the Wadham College men's group at Oxford in the autumn of 1986. Hard to recall now whether I went along out of simple curiosity or because I'd heard that the fastest way to a Wadham woman's heart was via an anti-sexist discussion forum, but it didn't seem so outlandish back then. This was the era when Andrea Dworkin was a disapproving presence on every female student's bookshelf and when a French guidebook directed tourists to Wadham to gaze at the "beautiful feminists" reclining on the lawns. So embedded were the new anti-sexist mores, college rumour told of a third-year who had trained himself not to get an erection with his girlfriend, thereby avoiding a physical state that was irredeemably aggressive.

It wasn't nostalgia that brought back these memories, but rather a glimpsed photograph of Alastair Campbell wearing an 80s-style T-shirt bearing, in bold capital letters, the slogan No More Page Three. Good for him and good for that campaign, which advanced this week with the decision by the Irish edition of the Sun to drop the famous topless picture in deference to what it called "cultural differences". But the Campbell snap and the response – tweeted jokes about the former spin guru's chest or urging him to get the rest of his kit off – confirmed both how rare and how open to ridicule are forays by men into the war against sexism.

That there is a battle to be fought is surely beyond doubt. Whether it's a prosecuting barrister branding a 13-year-old female victim of sexual abuse "predatory", or the ongoing death and rape threats against women who speak out on social media, all those who care about even basic notions of fairness or justice can see there is a momentous struggle to be joined. Yet men hesitate. Register the voices who rise up to object to these or any of the other instances, constant and ubiquitous, of sexism and misogyny and they overwhelmingly belong to women.

Perhaps that's inevitable. An attack on any group will be felt first and most keenly by that group: it usually falls to Jews, for example, to sound the alarm over antisemitism. But that rule is not universal. The backlash against the Home Office's "Go Home" vans, a hateful scheme now under investigation by the Advertising Standards Authority, has not been the exclusive preserve of immigrants, legal or illegal, or the descendants of immigrants. Even Nigel Farage denounced it.

But somehow men leave the heavy lifting against gender bias and gender hatred to women. The most charitable explanation is that men worry they cannot speak about this subject authentically, that their perspective is of less value than a woman's. Others fret they'll get it wrong, that they'll inadvertently say something that is itself sexist, thereby revealing that they too don't "get it" – so it's safer to say nothing. The diffidence of the men who took part in last week's #twittersilence was striking, several indicating that they were only "sort of" taking part.

Underlying all of this is that fear of ridicule, the suspicion that there is something funny about a man in a No Page Three T-shirt, or even about the simple act of calling himself a feminist. My remembered student experience is part of that, the notion that if a heterosexual man takes anti-sexism too seriously he'll end up emasculated, humourless and ideologically barred from expressing sexual desire – in other words, less of a man.

The result in what should be a universal movement for human equality is a big gap where the men should be. Of course the differences between sexism and racism are vast, but it's useful to recall the great civil rights struggles of 50 years ago all the same. That was an African-American movement from top to bottom, from its leaders to its grassroots, as it had to be. But white anti-racists were part of it. Scan the photographs of those freedom marches and there are white faces as well as black.

That was necessary, for the twin and bleakly simple reasons that white Americans were both the problem and an essential part of the solution: racism did not exist in the abstract, but in the hearts of white people and white-led institutions, and it was white people who held the power to change things. Martin Luther King and Rosa Parks prevailed because they not only raised the consciousness of black America, they moved and shifted white society too.

Again, the parallel is imperfect, but surely contemporary feminism has to engage men for similar reasons. Dip into the eye-opening @EverydaySexism feed on Twitter and you will see evidence of the most egregious discrimination – women assaulted and insulted as they go about their daily lives – almost all of it committed by men. At the risk of stating the obvious, progress requires more than the testimony of the woman told in a job interview that it'd be nice to have some "eye candy" around. It will also require men to stop saying it.

That means a change in men, but also perhaps in the struggle itself. For there is not just a gender gap on this issue. Wary as I am of pointing it out, there does seem to be a gulf separating the feminist conversation currently aired loudest in the public sphere and the kind of monotonous, grinding experience recorded by @EverydaySexism. It is the culture wars that grab media interest – a run of pop videos featuring topless women; proposed "modesty" wrappings to hide the covers of lads' mags; Jane Austen on bank notes; horrors on Twitter – yet it is the stubborn problems of unequal pay, low conviction rates for rape, workplace discrimination against mothers and, say, the need for statutory carer's leave, which probably speak more directly to the lives of women outside the media bubble.

For now, though, the challenge is for men to find their place – and to be welcomed – in a struggle that may be led by the women's movement but which is surely a human cause. We've tried sitting in silence – and it hasn't worked.

Twitter: @Freedland