Three years ago, me and my brother Ivor made a short film for Kick Racism Out of Football called The Y-Word, which attempted to challenge the acceptability of the chanting of the word Yid (and other antisemitic chants) across a variety of London clubs. Last week the Football Association condemned the use of the word by fans, and this week it's been announced that the Tottenham Hotspur Supporters' Trust is to poll members about the continued used of the chant.

I welcome both these developments – although not as much as I welcome David Cameron weighing in on the other side of the argument by backing the right to chant: from a campaigning point of view, that's a dream come true. But the debate involves a number of misconceptions. Primary among these is the idea that it's all about Spurs. It isn't.

Our film was sparked by the behaviour of a Chelsea fan who, sitting a few seats behind me and Ivor one Saturday, decided to upgrade the chant – regularly heard at Stamford Bridge whenever anything Spurs-related comes up – to a more pointed one of "Fuck the fucking Yids! Fuck the fucking Jews!". The chant, and various antisemitic tropes which always grow out of it – involving hissing to represent gas and celebratory references to Auschwitz – exists far beyond White Hart Lane: at Chelsea, Arsenal, Millwall, West Ham, even at Ajax, in Amsterdam.

However, there is of course a particular issue with Tottenham, some of whose fans passionately feel the Y-word is part of their identity and that their chanting of it is wholly positive. I respect and acknowledge that. But here are the reasons why it is good that this is being addressed.

First, Spurs fans often tweet me, forcefully, to say that historically the chant was a response to antisemitic abuse levelled at them. That's as may be; but truly, it doesn't matter who started it. The fact is that whatever its origins, their continuing use of the Y-word legitimises and sustains the racist abuse aimed at Spurs by other fans. It's a call and response dynamic, like many at football matches. So the more Spurs do it, the more it comes back, with menaces. Many Chelsea fans who I have challenged feel they are justified because "the Yids is what Spurs call themselves".

Second, any campaign aimed at stopping the chanting of antisemitic abuse at football matches can't then say: "But of course it's OK for this one set of fans because they mean it nicely." It's simply not workable.

Third, most Spurs fans are not in fact Jewish: the club's "Jewishness" is just a historical association with the area. It's doubtful that more than 5% of those in the ground at home games are actual Jews (only 0.4% of the UK is Jewish, so 5% is way above average). So the reclamation argument does not apply, unless it's OK for a race-hate word to be reclaimed by people who do not own it. The equivalent case would be a club in Brixton made up mainly of white fans adopting the N-word as their "badge of honour". This, I think, would be stopped fairly quickly.

The last point is the key one. Let me say it again. Yid is a race-hate word. It was daubed across the East End by Oswald Moseley's Blackshirts, along with the word Out. The only possible reason why a culture that has tried to dismiss other race-hate words to the margins of language would consider it acceptable is if the racism of which it is a part is somehow less offensive, somehow less significant, than other racisms. Which must be, I guess, what a lot of people consciously or unconsciously think – if it had been the N-word or the P-word, it wouldn't have got past David Cameron's lips.

This, at heart, is what this is about for me. It's not, in the end, to do with football, however much it was sparked by the miserable experience of 30 years sitting as a Jew in football grounds listening to huge groups of non-Jews bat an insulting word for my race back and forth in an endless slanging match. It's about trying to ensure that there is a level playing field around offensiveness. Ideally, I would be very uninterested in banning any words. But if the race-hate words used to denigrate black or Asian people are to be banned or considered unacceptable, then the same should apply for the vocabulary aimed at Jews.

Because the truth is, up till now, that vocabulary has not been considered equivalently offensive. A friend of mine – a Watford fan who heard his fellow fans chanting the Y-word aggressively at Spurs fans once – was told, on complaining to a steward: "I don't hear any racism." This is all our campaign has been trying to do: make the racism heard, above the chanting.