One of the occupational hazards of writing a column – apart from a slow but inevitable ballooning of the ego – is being told off for all the columns you haven’t written. There is a certain species of person who enjoys emailing columnists, brandishing how they haven’t written about Subject X as evidence of a) their moral shortcomings; b) an establishment cover-up; or c) rank hypocrisy. The model is columnist-as-Pez-dispenser: issuing regular and precisely calibrated condemnations of every evil in the world. Sometimes I feel like Bridget Jones at the book launch, where she starts by praising Salman Rushdie and ends up having to be nice about Jeffrey Archer to seem even-handed. (“Violence against men . . . is bad as well!”)

Which brings me to a question I have been asked recently: why haven’t I written about the Cologne attacks? Women reported a string of sexual assaults in the German city on New Year’s Eve, with many of the perpetrators said to be of “Arab or North African origin”. The story became inextricably linked to Europe’s debate about refugees.

There is now no doubt something awful happened that night. On 18 January, a 26-year-old Algerian asylum-seeker became the first person to be arrested over the sexual assaults; nearly a dozen other men have been arrested on charges of robbery. However, not all recent claims of migrant violence have stood up to scrutiny: a 13-year-old girl who claimed she had been kidnapped and raped by “Middle Eastern” migrants later retracted the story, saying she had invented it to avoid punishment for skipping school.

This fuzziness was my initial reason for not wading in. The reports were sketchy, in a language I abandoned after GCSEs half a lifetime ago, and from the start it was unclear if the attacks were perpetrated by existing migrants, new refugees, or even German citizens of Arab or North African origin. Besides, what did I have to offer beyond a straightforward condemnation?

As a feminist, I am opposed to all sexual harassment. It is a crude but effective weapon for making women feel that they are not welcome in public spaces and in public life more generally; I’ve been writing on and off for five years about internet abuse and how that puts off women from participating in discussions online.

Yet, for many, that simply won’t do. It is not enough to say that misogyny comes in many forms, and is depressingly ­universal across cultures and history. We have to cordon off the Cologne attacks; erect a little white tent around the crime scene and give thanks that we are safely outside it. Ah, how blissful it is, here on the outside, where the person most likely to kill a woman is her intimate partner, and where 85,000 women and 12,000 men are raped every year.

And that brings me to the other reason I didn’t want to write about the Cologne attacks. All the people who piously enquired as to whether I, as a feminist, had “anything to say” about them didn’t really care whether I did or not. They wanted me to say what they wanted to hear: that Muslims are uniquely sexist, and that letting in refugees from Muslim-majority countries will mean rolling back women’s rights and importing the worst excesses of sharia law to the streets of Coventry. Unless Western liberals wake up, Islamists will be chopping off hands outside Pret A Manger by 2018.

To put it politely, this is not the framing in which any reasonable conversation about women’s rights can happen. First, the terms are too vague: is the problem Muslims (all one billion of them)? Or men from specific countries? Or just “brown men” or “foreigners”? Without identifying the problem, there is little hope of a solution.

Then there is the musty undertone of paternalism mixed with white supremacy. When Dylann Roof stormed a historically black church in South Carolina, one of his grievances was that “you rape our women, and you’re taking over our country”. This formulation – “our women” – was also used by Tommy Robinson, formerly of the English Defence League, after the New Year’s Eve attacks. Reread the commentary on Cologne and count how much concern is expressed for migrant women, shackled for life to these attackers, or for the families that unaccompanied male migrants have left behind to live in poverty. You won’t find much. In this formulation, the problem is not that certain men are misogynist; it’s that the targets of their misogyny belong to someone else. To me, the unspoken coda to “You rape our women” is always “. . . and that’s our job”.

You can see this most clearly in the rhetoric of the self-described men’s rights activists, whose usual response to allegations of sexual assault is disbelief. (Their websites are full of accusations that women routinely lie about rape.) And yet, in the case of Cologne, they have become instant converts to #ibelieveher. Why? Because this allows them implicitly to reproach Western feminists for not seeming grateful enough to men for allowing them the freedoms they currently enjoy. In this way, women’s ability to walk safely in public is cast not as a fundamental human right, but as a special privilege, nobly granted to them by European men.

At the women’s charity where I volunteer, there is a poster that says: “She’s someone’s daughter, sister, mother.” All the qualifiers are crossed out, leaving the simple statement: “She’s someone.” Each of the women attacked in Cologne was someone. What matters is not that “they” attacked “our” women, but that the patriarchy and male violence endemic across the world took a particular and extreme form that night in Germany. And so I parry the accusation of hypocrisy against me with one of my own: if your interest in misogynist violence starts and ends with Cologne, you don’t really care about women at all.