Prime Minister Stephen Harper is "dumb" to say that wearing a niqab is anti-women, according to Charles Taylor, one of the men behind Quebec's landmark Bouchard-Taylor Commission on Reasonable Accommodation.

"How dumb do you have to be to not see that a remark like that in the present context [of heightened tension around the issue of Islamic terrorism] is inflaming," Taylor said in an interview on CBC News Network's Power and Politics.

"Now he's trying to row away from it but I wonder if he's trying to raise his vote in Quebec because his polls did go up there after that [remark]."

Taylor was referring to a comment made by the Prime Minister on March 10 during question period when he said the niqab, a face covering worn by some Muslim women, is "rooted in a culture that is anti-women."

"You have to be very careful that you don't stigmatize Islam in general. The prime minister is playing along with an unfortunate tendency, an Islamophobic tendency, in North America as a whole. That is a direct danger to our security and it's terrible for our society," Taylor said.

The prime minister is taking a page out of the Parti Québécois playbook by banning the niqab during citizenship ceremonies, Taylor added. That party, under the leadership of former premier Pauline Marois, tried to enact a so-called secular values charter, which would have banned religious symbols in the public service.

Marois lost to Liberal Philippe Couillard in the 2014 Quebec general election.

Taylor, a well-known political philosopher, and a federalist, and Gérard Bouchard, a sovereigntist historian, co-authored their reasonable accommodation report in 2008. The report was drafted after months of testimony from immigrants and Franco-Quebecers alike.

In the report, Taylor stressed that Quebecers need to demonstrate an "openness and generosity of spirit" for minorities. The report also recommended, among other things, that students who wish to wear religious symbols in class, such as the hijab, kippah or turban, should be able to do so.