As a person who writes about women's issues, I am constantly being told that Islam is the greatest threat to gender equality in this or any other country – mostly by white men, who always know best. This has been an extraordinary year for feminism, but from the Rochdale grooming case to interminable debates over whether traditional Islamic dress is "empowering" or otherwise, the rhetoric and language of feminism has been co-opted by Islamophobes, who could not care less about women of any creed or colour.

The recent blanket coverage of the "gender segregation on campus" story was a textbook case. This month Student Rights, a pressure group not run by students, released a report vastly exaggerating a suggestion by Universities UK that male and female students might be asked to sit separately in some lectures led by Islamic guest speakers. Many Asian women's groups and individual Muslim feminists joined the subsequent protests, sometimes taking personal risks to do so. Unfortunately, rightwing commentators and tabloids seized upon the issue to imply that Islamic extremists are taking over the British academy.

Never mind that it wasn't strictly true, the non-controversy spread to every level of government. Labour MP Chuka Umunna declared: "A future Labour government would not allow or tolerate segregation in our universities." Even the prime minister stepped into the debate, saying the proposed guidelines, which have since been withdrawn, were "not the right approach". The elite all-male Oxford club of which both he and the chancellor were members was presumably the perfect approach.

I have spent weary weeks being asked to condemn this "policy of gender segregation" by "Islamic extremists", despite the fact that no such policy exists. Of course, I condemn all sexism within the academy. I condemn segregated drinking societies and the under-representation of women at the top levels of academia. I condemn rape culture on campus, traditions like "seal clubbing" and "slut dropping" where male students are encouraged to sexually humiliate their female classmates. If I've enough breath left, I'll condemn the suggestion that guest lecturers be allowed a segregated audience for religious reasons.

Structural sexism does take place every day in our universities, as it does in our offices, shops and homes – and we should oppose it everywhere. But demanding that feminists of every race and faith drop all our campaigns and stand against "radical Islam" sounds more and more like white patriarchy trying to make excuses for itself: "If you think we're bad, just look at these guys."

It's the dishonesty that angers me most. It's the hypocrisy of men claiming to stand for women's rights while appropriating our language of liberation to serve their own small-minded agenda. Far-right groups like the English Defence League and the British National party rush to condemn crimes against women committed by Muslim men, while fielding candidates who make claims like "women are like gongs – they need to be struck regularly".

Some of their members tell me that since they are standing against the sexism of Muslim barbarians, as a feminist I should be on their side. When I disagree, I am invariably informed I deserve be shipped to Afghanistan and stoned to death.

Horror stories about Muslim misogyny have long been used by western patriarchs to justify imperialism abroad and sexism at home. The Guardian's Katharine Viner reminds us about Lord Cromer, the British consul general in Egypt from 1883. Cromer believed the Egyptians were morally and culturally inferior in their treatment of women and that they should be "persuaded or forced" to become "civilised" by disposing of the veil.

"And what did this forward-thinking, feminist-sounding veil-burner do when he got home to Britain?" asks Viner. "He founded and presided over the Men's League for Opposing Women's Suffrage, which tried, by any means possible, to stop women getting the vote. Colonial patriarchs like Cromer … wanted merely to replace eastern misogyny with western misogyny." More than a century later, the same logic is used to imply that misogyny only matters when it isn't being done by white men.

I am not writing here on behalf of Muslim women, who can and do speak for themselves, and not all in one voice. I am writing this as a white feminist infuriated by white men using dog-whistle Islamophobia to derail any discussion of structural sexism; as someone who has heard too many reactionaries tell me to shut up about rape culture and the pay gap and just be grateful I'm not in Saudi Arabia; as someone angered that so many Muslim feminists fighting for gender justice are forced to watch their truth, to paraphrase that fusty old racist Rudyard Kipling, "twisted by knaves to make a trap for fools".

We are the fools, if we believe that accepting aggressive distinctions between nice, safe western sexism and scary, heathen Muslim sexism is going to serve the interests of women. The people making these arguments don't care about women. They care about stoking controversy, attacking Muslims and shouting down feminists of all stripes.

For decades, western men have hijacked the language of women's liberation to justify their Islamophobia. If we care about the future of feminism, we cannot let them set the agenda.

This article was amended to draw attention to the fact that many Muslim and Asian women were involved in the "gender segregation" protests