On the tenth anniversary of the 9/11 attacks in New York 18 months ago, I found myself in front of the American embassy in London being heckled by Muslims Against Crusades (MAC) activists, and praised by English Defence League members. While our motley group held up placards aimed at Anjem Choudhary’s group, a small group of religious Muslims (sporting beards and hijabs) arrived and set up a table with biscuits and Union flags. Then they started handing out flyers explaining why MAC were distorting Islam. The MAC activists were momentarily confused. Then they started heckling them too. The EDL almost felt ignored.

A week earlier, a friend had called to express his anger that MAC were planning a demonstration that day, and wanted to show that not all Muslims agreed with them. So we organised our own counter-demonstration and made our own banners. EDL members, who had come to throw abuse at MAC, were so pleased by our presence they wanted to take pictures with us (we declined). It was a bizarre day, but we also decided that this had to become a regular thing.

One thing we know – at least one of the key suspects in the Woolwich attack on Lee Rigby was a member of Anjem Choudhary’s inner circle. Before Muslims Against Crusades they called themselves Islam4UK and before that, Al-Muhajiroun. During the Danish Cartoons controversy of early 2006 they held an infamous demo in London calling for the beheading of "those who insult Islam". They are banned from almost every mosque in the country and ostracized by almost every British Muslim community group.

It was Al-Muhajiroun that sparked the formation of the EDL after a protest in Luton as British soldiers marched by. It is always Al-Muhajiroun that pull publicity stunts designed to inflame the media: burning poppies on Rememberance Day, marching on Wootton Basset and so on.

This makes it all the more remarkable that left-wing groups don’t mobilise against these religious extremists as they do against the far-right. Anti-fascists who happily march against the BNP or EDL rarely show that level of commitment against Anjem Choudhary’s group. Why? There even seems to be a reticence to admit that the EDL feeds off Muslim extremists. I’m not saying the Left embraces or even excuses away these clerics, but this strange reticence across the Left not only allows them to fester, but has other consequences.

For a start, taking on Muslim extremists denies a space for the English Defence League to flourish. Sure, many EDL members are looking for any excuse to express their racism, but the far-right group also draws in recruits who don’t consider themselves racist but want to oppose religious extremists. Having a range of Muslims and non-Muslims publicly opposing the likes of MAC is also the best way to silence rightwing critics who use the latter to demonise and generalise about all Muslims.

Secondly, building a broad alliance against religious extremists would also shed light on other unsavoury groups. In recent years university campuses have invited xenophobic preachers such as Haitham al-Haddad, Kamal El Mekki and others, despite their extreme views on women, homosexuals and integration. Islamic Societies at Kingston, London South Bank and London Metropolitan universities have all come under deserved criticism, while the UK-based Islamic Education & Research Academy has a whole roster of xenophobic preachers. These groups and socities shouldn’t be banned – after all we must value free speech – but they should be actively opposed by left-wingers who care about the proliferation of bigotry in our society. Otherwise its painfully obvious that we are not applying our principles consistently.

The third and perhaps the most important reason we must do this is to support more moderate Muslims. Take imam Usama Hasan for example. Two years ago he received death threats after giving a talk at his mosque backing evolution and women’s rights. After a futile struggle he was eventually forced out by extremists who made his life hell. He told me he isn’t alone, and that extremists preachers who are less publicity-hungry than the likes of Anjem Choudhary and Abu Hamza are diligently working to take over mosques and oust more moderate Muslims who preach integration. By ignoring Muslim extremists we also abandon the likes of imam Hasan, who want to preach a more incusive version of Islam.

In 2010, after a poppy-burning stunt by Al-Muhajiroun, campaigners from Hope Not Hate adopted a "plague on both their houses" approach, with the approval of their members. But within days they were branded as "Islamophobes" by certain other anti-fascists and came under a surprising amount of criticism. This has to stop. In the aftermath of Woolwich, if we are serious about undermining the EDL and tackling all forms of bigotry, we cannot ignore religious extremism.