You can tell that speakers are preparing to say something scandalous when they assert that "militant atheists" are the moral equivalents of the religious militants that so afflict humanity. Trevor Phillips, whose flighty management of the Equality and Human Rights Commission is becoming a scandal, was no exception when he announced last week that British believers were "under siege" from "fashionable" atheists.

If his claim that "people who want to drive religion underground are much more active, much more vocal" contained a jot of truth, we would have read the following stories in the days after his intervention.

• Inflamed after reading an acerbic passage by Richard Dawkins, "fashionable" Belfast atheists decide to lay siege to Catholic homes in the Short Strand area of the city. They terrify its residents and attack the police with petrol bombs and fireworks. (As it was, the riots were the work of Belfast Protestants motivated by a hatred of Catholicism. They were met by Republican IRA "dissidents", filled with an equal hatred of Protestantism.)

• "Vocal" Iraqi secularists decide that they want to drive the Shia Muslims in Baghdad underground. They ignite bombs in a Shia market during its busiest time of the week and a mosque, killing 40 in all. (As it was, the murders were the work of al-Qaida in Iraq, which regards Shia Muslims as heretics and was determined to demonstrate again that no one is as murderously "Islamophobic" as Islamists are.

• Free-thinking Americans decide they have had enough of religious leaders laying down the law. They descend on the New York State Senate and heckle and jostle a woman rabbi as she tries to influence a debate on gay marriage. (As it was, the heckling and jostling was done by Orthodox Jews, who said the rabbi had no right to speak for Judaism because she was a lesbian.)

Since the end of communist-inspired persecutions in all the old socialist countries bar China and North Korea, religious hatred has become unique among the prejudices. Overwhelmingly, it is directed by the religious against the religious. Domineering believers threaten members of their own faith when they break taboos by experimenting with new thoughts and ways of living. Or they engage in sectarian conflicts with other religions.

Trevor Phillips's attack on "fashionable" atheists for exercising their right to speak their minds shows he does not begin to understand modern sectarianism. From his ignorance flows a cowardly refusal to face down those who would bully and harass others, as a story that deserves more attention than it has received shows.

British Asians, secularists and Liberal Democrat and Labour politicians have been trying for years to persuade the government to tackle caste discrimination. They have had no success because the treatment of untouchables is one of the great unmentionables of British politics. They are certainly the victims of a form of religious prejudice – the sanction for the oppression of lower castes in a pre-ordained hierarchy comes from Hindu creation myths. Yet caste prejudice does not fit easily into established views of how discrimination works, because caste divisions exist among Sikhs, Muslims and Christians whose families came from the sub-continent, as well as Hindus.

Faced with the prospect of confronting the prejudices of core supporters, the Labour government preferred holding on to seats to living by liberal principles and backed away from extending anti-discrimination law to cover caste. With Labour gone, campaigners for just treatment for tens of thousands of British Asians have a glimmer of hope.

They are trying to persuade the coalition to take seriously a study of bullying and harassment conducted by the National Institute of Economic and Social Research. It is a dispiriting read – little more than a list of pointless cruelties. The Indian supervisor of an NHS worker discovers that he is from a lower caste and makes his life such a misery he becomes ill under the pressure and is suspended; a social services care worker refuses to help an elderly woman wash herself because the old lady is from a lower caste and so it goes on through dozens of examples.

The casual observer of British politics might have thought that a voluble quangocrat, who is always willing to fill empty airtime with heart rending cries for greater equality, would have denounced caste prejudice with unembarrassed vigour. For once, however, Phillips is silent. A search of the Equality and Human Rights Commission records shows that it ignores caste discrimination in Britain.

When I phone its press office to ask why, its public relations officers fail to return my calls. Without hearing his side of the story, I can only guess that Phillips does not like admitting that ethnic minorities as well as white people are capable of prejudice. He may worry, too, that an honest stance would require him taking on religious lobbyists, such as Hindu Council UK, which questioned "the existence of caste discrimination in the UK" on Friday and claimed that the issue was being manipulated by Christians eager to convert Hindus from their faith.

In this instance, Phillips not only refuses to campaign for the disadvantaged, but is alleged to hinder those who do. Keith Porteous Wood, of the National Secular Society, said he had been "no help at all. Advances we have made have been despite him, not because of him". The normally mild-mannered Lib Dem peer Lord Avebury said that "Phillips has played an ignoble part in suppressing this issue."

From the leftish point of view there is no good ground for keeping Phillips in post. The liberal left ought to know that caste discrimination is a greater evil than class discrimination because, whatever an individual's accomplishments, he or she can never escape from the hereditary curse. It ought also to feel a tinge of shame that when the victims of prejudice try to start a new life by coming to Britain, they find that the old prejudice follows them here – and that the Equality and Human Rights Commission will do nothing about it.

From a Tory standpoint, the case against Phillips is as easy to make. When the government has had to raise taxes and cut spending, what purpose is served by carrying on spending taxpayers' money on Phillips? With a bit of luck, left and right will soon agree that removing him from an office he is unfit to hold is a "fashionable cause" everyone can support.