David Cameron was accused of playing into the hands of rightwing extremists today as he delivered a controversial speech on the failings of multiculturalism within hours of one of the biggest anti-Islam rallies ever staged in Britain.

Muslim and anti-fascist groups questioned the prime minister's judgment and sensitivity to the issues, saying he had handed a propaganda coup to the hard-right English Defence League as 3,000 of its supporters marched through Luton chanting anti-Islamic slogans.

Some of crowd were jubilant, saying that Cameron "had come round to our way of thinking". Paul Bradburn, 35, from Stockport, said Cameron was "coming out against extremism".

He added: "The timing of his speech is quite weird as it comes on the day of one of the biggest EDL demos we've ever seen. If he wants to start sticking up for us, that's great."

Matt, 16, a school pupil in Birmingham who was at the march said: "He believes what we believe to some extent."

Downing Street issued a robust defence saying the prime minister was "absolutely unapologetic".

A spokeswoman said the speech had been "in the diary for months". She added: "The idea that he would be blown off course on an issue as fundamental as this by the English Defence League is ridiculous and extraordinary."

Cameron told the Munich Security Conference, attended by world leaders, that state multiculturalism had failed in this country and pledged to cut funding for Muslim groups that failed to respect basic British values.

He blamed the radicalisation of Muslim youths and the phenomenon of home-grown terrorism on the sense of alienation that builds among young people living in separate communities and the "hands-off tolerance" of groups that peddle separatist ideology.

Just a few hours later, EDL leader Stephen Lennon told the crowd they were part of a "tidal wave of patriotism" that was sweeping the UK.

Activists, some wearing balaclavas and others waving English flags, chanted "Muslim bombers off our streets" and "Allah, Allah, who the fuck is Allah". EDL supporters from Newcastle, Scotland, London, West Yorkshire and Sheffield joined Luton-based supporters. There were also flags representing German, Dutch and Swedish Defence Leagues.

Nick Lowles, director of anti-extremist group Hope Not Hate, said the timing of Cameron's remarks had allowed EDL members to claim the government was on its side.

"The prime minister's comments were unhelpful. On a day when extremist groups of varying persuasions were descending on Luton, his words were open to misinterpretation at best, and at worst were potentially inflammatory.

"Whatever the intention, the timing of this speech has played into the hands of those who wish to sow seeds of division and hatred."

Mohammed Shafiq, chief executive of the Ramadhan Foundation, said Cameron had handed a "propaganda coup to the EDL and their extremists".

Labour also weighed in, attacking the timing of the speech and rejecting implications from Cameron that it had failed to address issues of Islamic extremism and the complex issues of multiculturalism during 13 years in office.

Former home secretary Jack Straw said it was "ill-timed" and "ill-judged". Former Labour minister Margaret Hodge said: "This is a hugely difficult area. I agree that there are some areas where we need strong assimilation – speaking English and abiding by British law.

"But Cameron appears to suggest we can impose a much wider assimilation with British values and the danger is that this approach will perversely entrench those separate identities that he wants to meld.

"The state has to be very cautious in using its power to mould cultural values. Comparisons with far-right groups on the day the EDL is mounting a demonstration is needlessly provocative.

"It is not merely the Muslim community's responsibility to combat extremism; we all have responsibilities, particularly to ensure that minority communities do not feel excluded."

As the row over the prime minister's remarks intensified, Tory co-chairman Baroness Warsi called for an apology from shadow justice secretary Sadiq Khan after he accused Cameron of "writing propaganda for the EDL".

"For Sadiq Khan to smear the prime minister as a rightwing extremist is outrageous and irresponsible. David Cameron has made it clear that he wants to unite Britain around our common values and he has done so in measured language," she said. "It is right that we make it clear: extremism and Islam are not the same thing.

"And, as David said, it's important to stress that terrorism is not linked exclusively to any one religion or ethnic group."