No one actually comes out and directly says "I hate Muslims" – at least, not on the liberal dinner party circuit that was the target of Lady Warsi's speech. Conversations generally begin with the sort of anxieties that many of us might reasonably share: it cannot be right for women to be denied access to education in some Islamic regimes; the use of the death penalty for apostasy is totally unacceptable; what about the treatment of homosexuals? The conversation then moves on to sharia law or jihad or the burqa, not all of it entirely well informed. Someone places their hands across their face and peers out between their fingers. Another guest giggles slightly. Someone inevitably mentions 9/11. Later, guests travel home on the tube and look nervously at the man in the beard sitting opposite.

The problem Warsi identifies is the problem of slippage. What can begin as a perfectly legitimate conversation about, say, religious belief and human rights, can drift into a licence for observations that in any other circumstance would be regarded as tantamount to racism. Like the 19th-century link between anti-Catholicism and racism towards the Irish, one can easily bleed into the other.

"I treat the Islamic religion with the same respect as the bubble-gum I scrape off my shoe," suggested one contributor to the website of the Richard Dawkins Foundation for Reason and Science, in response to Warsi's speech. Another offered the following charming observation: "I don't care what the good or bad Baroness has to say about anything at all. I give her no credence nor voice. She is a person of faith so in my book a skinwaste." I cannot think of a single other group in our society about whom such vile remarks would be in any way socially acceptable. And OK, these are comments whose surface grammar is about Islam and religion. Nonetheless, the level of invective is very obviously personal.

The worst sort of dinner party bigot may talk about Islam as a faith but – nod, nod, wink, wink – we all know what they mean. Just as we know that when the British National party celebrates the Christian heritage of this country it is using Christian as code for "not Muslim". In many cases, Muslim can easily become a euphemism for brown. Prejudice like this is a dance of the seven veils that allows just enough insight into one's true meaning and just enough deniability.

None of which is to silence any sort of attack upon religious faith per se. Polly Toynbee was right that "Muslims must accept the right of others to criticise religions without smearing any critic as a racist". While this is undoubtedly the case, the flip side is also true: that the attack upon Islam as a religious creed can easily become an excuse for an attack upon one ethnic group. It is vital that we find a better way of charting a course between these two dangerous snares.

One of the tests for flushing out prejudice from robust but legitimate critique is the extent to which complexity is allowed to enter into the picture. The dinner party bigot may never have been to a mosque or read the Qur'an, but he already knows what he thinks. Life is always simple for the prejudiced. Indeed, the very point about a prejudgment is that it is a conclusion reached before the complexity of the world is allowed to make any difference. The facts are forced to fit a pre-formed picture. What about Islam's historic contribution to science? What about the significant number of women who have become heads of government or heads of state in Muslim countries – Turkey, Pakistan, Indonesia, Bangladesh? So much of this is ignored in the rush to find Islam guilty of crimes against humanity. The good critic, on the other hand, doesn't need to oversimplify in order to make their point. And with so much at stake, rhetorical flamboyance needs to be handled with care.

The other difference between robust critique and what is tantamount to bullying has to do with the power relations between those involved. The Muslim community in this country is generally more socially disadvantaged and has less access to the levers of power. British Muslims do worse at school than any other faith group, they are more likely to be unemployed and live in poorer housing. It is generally from communities such as this that the prosperous and the powerful find their scapegoats.

This is also why the growing idea that there is in this country such a thing as Christianophobia – an equivalent to Islamophobia – is such total nonsense. Following Warsi's comments, the usual suspects of the Christian right have waded in with another rendition of "what about us?" What about those nice Christian B&B owners who have just been fined for sticking to their sincerely held beliefs about gay couples not sharing a bed under their roof? But the power relations here are altogether different. With bishops in the House of Lords by right, with the monarch being head of the Church of England, with the long history of Christianity shaping our values and culture, Christians are not a persecuted minority, however much they may feel misunderstood.

Islamophobia is the moral blind spot of 21st-century Britain. Warsi got the emphasis wrong in placing responsibility for this at the door of hostility to religion per se – though the tone of that debate is sometimes a proxy for much uglier sentiments. David Hume was right: reason is a slave to the passions, especially our darker ones. The real driver is that otherwise polite people have given themselves permission to be racist. Now is the time to disturb the cosy rules of the dinner party and speak up against the bigots. There may well be a row. You might not get invited back. But so what?

• This article was amended on 7 February 2011. The original referred to the significant number of women who have become heads of state in Muslim countries. This has been corrected.