Conservative MPs did not seem too concerned as Lady Warsi sounded the alarm that their party was institutionally riven with Islamophobia.

“I don’t really believe we have that big a problem,” said one former minister, who seemed surprised to be asked about the issue.

When 14 members of the party were suspended for allegedly Islamophobic comments on a Facebook group supporting Jacob Rees-Mogg, the response was to try to deflect attention on to Labour.

“Islamophobes have no place in the Tory party and it is encouraging that we have acted swiftly, unlike the socialists,” Rees-Mogg pronounced.

From much of the rest of the party, there has been a deafening silence. Brandon Lewis, the Conservative party chairman, responded to Warsi on Twitter accusing her of “missing out on key facts” about the party’s “clear process” and “swift action”.

Facebook Twitter Pinterest Brandon Lewis, the Conservative party chairman, hit back at Warsi. Photograph: Ian Forsyth/Getty Images

The effect has been to make Warsi – a Muslim woman who has personally suffered vicious abuse – look like she is fighting a lonely battle to get her party to acknowledge that the problem goes deeper than a few bad apples.

Meanwhile, Lewis has been tweeting sympathetically to a Labour MP complaining about antisemitism and criticising Jeremy Corbyn for not doing more. And there are few signs that the Conservative leadership is heeding warnings that this gleeful taunting of Labour could backfire.

Mohammed Amin, the chairman of the Conservative Muslim Forum, said he had been telling Lewis that more action needed to be taken on Islamophobia for a long time but Conservative headquarters had made clear there would be no independent internal inquiry into anti-Muslim hatred.

“The party leadership is still in that stage of ‘keeps wishing that the issue would go away’,” he said. “Every day when local councillors issue social media tweets denigrating Muslims and sometimes MPs retweet these things … essentially, every time things like that happen, the position of the Conservative party with British Muslims takes a step backwards. It is electorally damaging to the party.”

He said the party’s approach to the Muslim community had “gone backwards”, including Zac Goldsmith’s London mayoral campaign, Boris Johnson’s comments about women in burkas resembling letterboxes and Theresa May’s post-Brexit attempt to woo Ukip voters.

“Denial is the first way that most organisations react to accusations of this kind,” Amin said. “It takes a lot for longstanding Conservatives to rock the boat. The party has a long history of unity rather than openly arguing its case. But Baroness Warsi, Lord Sheikh and myself have done this because we care about the party and we care about the country.”

With only a few Conservative voices speaking out publicly on the issue, the party does not yet seem to have reached the same stage of visible political crisis as Labour has on antisemitism, following Jewish MP Luciana Berger’s decision to quit the party in protest at its failure to do enough to tackle antisemitic abuse.

But one Conservative Muslim parliamentarian, who did not want to go on the record, said they felt the problem of Islamophobia among the party’s rank and file might be just as widespread but more under the radar, especially since the party had far fewer activists and an older support base who were less likely to be active on social media.

While many Conservatives are reluctant to inflame the issue by speaking out, Anna Soubry, the chair of the all-party parliamentary group on British Muslims, who quit as a Tory MP over Brexit, said she had witnessed the leadership’s unwillingness to grip the problems caused by a drift to the right.

She said she would not go so far to say that the party was “institutionally Islamophobic” but there were too many examples where people with anti-Muslim views had not been booted out.

“This unwillingness to grasp the problem and root it out comes from a lack of leadership and facing up to the fact there are a significant number of Conservative members with publicly stated views that are odious and offensive and they have no place in the modern Conservative party,” she said.

“Brandon Lewis and Theresa May will not face up and deal with the fact that the party has drifted to the right and has been infiltrated. The problem is that Brandon just won’t grip the problem and whilst I don’t doubt he abhors Islamophobia and other forms of racism, he is unwilling to take swift and efficient action to expel people.”

Matthew McGregor, the campaigns director of the anti-racist charity Hope Not Hate, said there were signs that the Conservatives had “wilful blindness” about Islamophobic views in their rank and file. The organisation’s research found that 49% of Conservative voters see Islam as a threat to the British way of life and 47% believe the false conspiracy theory that there are areas where sharia law dominates and non-Muslims cannot enter.

“It does seem that more people at the core of the Labour party are finally understanding left antisemitism as a problem in itself – not just electorally – and are speaking out more forcefully,” McGregor said.

“There’s a way to go, and actions will speak louder than words, but the ground is starting to shift. It’s not clear that the Conservative leadership understand the scale of the challenge they face with Islamophobia in the Tory party. They need to acknowledge the problem if they’re going to tackle it.”

He said one approach on both sides that seriously needed to be avoided was any tribal attempt to divert attention from complaints about racism by pointing to another party’s record. “When both Labour and the Conservatives have serious issues with racism within their ranks, it has been grim to see each side trade blows over who is worse. Political point-scoring over the scale of the other party’s issues is just gross.”

Despite the temptation for politicians to score points, Sunder Katwala, the director of the British Future pressure group, said all parties should be listening hard when members of a certain faith said they were being made to feel uncomfortable.

“If you are a party that wants to govern, the test of you as a party is that no one who has got your values should feel a tension with their identity, faith and background,” he said. “It is definitely true that many Jewish people feel that about Labour. And the Conservatives should be listening to the Muslim members of their own party who feel there is an issue that needs a proactive stance, not just throwing out people once evidence comes to light.”