It's a crude reaction but it's the first one I had on hearing that the Swiss had voted to ban the building of minarets on mosques – the same reaction I have to the increasingly-frequent stories like it: how would I feel if this were not about them, but us? How, in other words, would I react if this latest attack were not on Muslims but on Jews?

It's crude because no two situations are ever exactly the same, and Muslims and Jews have different histories – in Switzerland and everywhere else. But it's useful, allowing the testing of any proposition against an almost instinctive yardstick of decency.

So how would I react if the Swiss voted to restrict the way synagogues are built? With horror, of course. Indeed, the mere hint of such a proposal in the heart of Europe – given the blood-soaked history of the 20th century – would send a shudder down the collective spine. That reaction alone would tell me that, on this proposal, there was only one decent place to be – against it.

Or take Jack Straw's campaign against the niqab in 2006. He and his supporters made what they hoped was a subtle, nuanced case against women wearing the full veil, but my first thought was much simpler. What if a government minister told ultra-orthodox Jewish men that, in their full beards, it was hard to tell them apart, or that he disliked the custom that commands ultra-orthodox Jewish women to cut off their hair, covering their heads with either a wig or a hat? No matter how subtle or nuanced his reasons, I would feel that this was, at best, an act of bullying directed at a vulnerable minority or, at worst, the first step towards something much more menacing.

I'm clearly not the only who thinks this way, submitting proposed anti-Muslim actions to an informal "Jewish test". It seems the proponents of the minaret ban proceeded the same way. According to Tariq Ramadan, the initial target of the Union Démocratique du Centre (UDC) campaign was due to be the Islamic method of animal slaughter – until the UDC realised that Jews, who also rely on ritual slaughter to produce kosher food, would immediately feel threatened. So they moved onto minarets, apparently confident that they had found an issue with few Jewish resonances (synagogue buildings rarely have an impact on the skyline and are often pretty inconspicuous).

Of course, not everyone who voted yes on the Swiss referendum was some Islamophobe or racist. Some, as Joan Smith argues, had wholly admirable secularist motives. Indeed Smith makes a strong case for holding a larger discussion about the role of religion in public life and expresses frustration that we cannot seem to have a mature conversation about, for example, which symbols belong in public and which don't. I would welcome that conversation. I think we should have it. But here's a suggestion. Let's make sure that, for once, it doesn't start with the Muslims.

For example, here in Britain, we should have that debate about faith schools. But let's begin with a proposal to close down the 7000 Christian schools in the state sector – and then move onto the Muslim schools. Or do I sense that there will be rather fewer takers for that conversation?

What passionate secularists and atheists need to understand is that what seems to outsiders like a religious affiliation is, for many millions, only partly about faith. It's often partly, even largely, about identity. How can I be so sure that's true of Muslims? Because I know it's true of Jews.

Which leads me to guess that the minaret ban will have one consequence its advocates did not predict. I reckon there'll be rather more Swiss Muslims going to the mosque this week than there were last. That's how people react when they're threatened. Don't ask me how I know.