OTTAWA — Conservative Leader Andrew Scheer tried to dismiss accusations Saturday that he and his party have not done enough to distance themselves from groups that advocate violence against the Muslim community.

Speaking at the annual Manning Networking Conference, the Tory leader was asked by someone in the audience why he was unwilling to condemn Islamophobia.

WATCH: Scheer tackles Islamophobia, vows to balance federal budget at Manning Conference.

"I don't believe that's true," Scheer responded, noting that the House of Commons unanimously passed a motion earlier this week condemning the terrorist attacks against two mosques in New Zealand and the need to "confront hatred, Islamophobia, and white supremacy, in all their forms."

"I reject anyone who would, you know, who would speak out based on Islamaphobic principles whether or not, you know, that's somebody who is trying to lump all people of the Muslim faith in together, or whether it is people who are trying to antagonise elements of the society to have a more negative reaction to those who practice that faith," Scheer told the crowd of Conservatives, peppering his comments with uncharacteristic "ums" and "ahs."

To me the important thing is to speak out against those who would in any way give oxygen or space to those who are trying to promote one group of people over the other.Conservative Leader Andrew Scheer

"To me the important thing is to speak out against those who would in any way give oxygen or space to those who are trying to promote one group of people over the other," he said.

Scheer was criticized this week for failing to do just that.

The president of the Canadian Labour Congress, Hassan Yussuff, told reporters Thursday that Scheer should be held responsible for failing to condemn yellow-vest protesters who showed up in Ottawa in February to demonstrate against pipeline delays while spewing anti-immigrant rhetoric.

"For the leader of the Opposition not able to condemn them ... and stand on a stage with someone who is associated with hate and racism and somehow to not acknowledge that? ... That is a fundamental, I think, failing on his part, and I condemn his behaviour," Yussuff said at an event outlining efforts by the union movement to combat Islamophobia and the steady rise in hate crimes targeting Muslims.

"I don't think he should be excused for it, because he should know better, and if he doesn't, something is fundamentally wrong with his leadership, in my view," Hassan said.