If tomorrow, God forbid, I were to cause the death of an innocent man with my car, minutes after sending a series of texts on my mobile phone, I’m guessing I’d spend the rest of my life riddled with guilt. What I wouldn’t do is go on television and lay the blame for my subsequent 12-week imprisonment at the door of . . . wait for it . . . the Jews. Yet that’s what the Labour peer Nazir Ahmed did in April 2012 – less than five years after causing a car crash on the M1 in which Martin Gombar, aged 28, was killed.

“My case became more critical because I went to Gaza to support Palestinians,” he says to his Pakistani interviewer in Urdu, in a video recording obtained by the Times. “My Jewish friends who own newspapers and TV channels opposed this.” The judge who put him behind bars, Lord Ahmed claims, was appointed to the high court after helping a “Jewish colleague” of Tony Blair’s during “an important case”.

To claim that your jail sentence for dangerous driving is the result of a Jewish plot is bigoted and stupid. The peer has since been suspended from the Labour Party and forced to stand down as a trustee of the Joseph Interfaith Foundation. I’m not sure how many “Jewish friends” he has left – if, that is, he had any to begin with.

Full disclosure: I know Lord Ahmed and have defended him in the past. In 2007, he flew out to Sudan to help free the schoolteacher Gillian Gibbons from the clutches of the odious Islamist regime in Khartoum. In 2009, an Appeal Court judge noted how the peer had “risked his life trying to flag down other vehicles to stop them colliding with . . . his car”. He is not a latter-day Goebbels. But herein lies the problem. There are thousands of Lord Ahmeds out there: mild-mannered and well-integrated British Muslims who nevertheless harbour deeply anti-Semitic views.

It pains me to have to admit this but anti-Semitism isn’t just tolerated in some sections of the British Muslim community; it’s routine and commonplace. Any Muslims reading this article – if they are honest with themselves – will know instantly what I am referring to. It’s our dirty little secret. You could call it the banality of Muslim anti-Semitism.

I can’t keep count of the number of Muslims I have come across – from close friends and relatives to perfect strangers – for whom weird and wacky anti-Semitic conspiracy theories are the default explanation for a range of national and international events. Who killed Diana and Dodi? The Mossad, say many Muslims. They didn’t want the British heir to the throne having an Arab stepfather. What about 9/11? Definitely those damn Yehudis. I mean, why else were 4,000 Jews in New York told to stay home from work on the morning of 11 September 2001? How about the financial crisis? Er, Jewish bankers. Obviously. Oh, and the Holocaust? Don’t be silly. Never happened.

Growing up, I always assumed that this obsession with “the Jews” was a hallmark of the “first-generation” immigrants from the subcontinent. In recent years, I’ve been depressed to discover that there are plenty of “second-generation” Muslim youths, born and bred in multiracial Britain, who have drunk the anti-Semitic Kool-Aid. I’m often attacked by them for working in the “Jewish owned media”.

The truth is that the virus of anti-Semitism has infected members of the British Muslim community, both young and old. No, the ongoing Israel-Palestine conflict hasn’t helped matters. But this goes beyond the Middle East. How else to explain why British Pakistanis are so often the most ardent advocates of anti-Semitic conspiracies, even though there are so few Jews living in Pakistan?

It is sheer hypocrisy for Muslims to complain of Islamophobia in every nook and cranny of British public life, to denounce the newspapers for running Muslim-baiting headlines, and yet ignore the rampant anti-Semitism in our own backyard. We cannot credibly fight Islamophobia while making excuses for Judaeophobia.

To be honest, I’ve always been reluctant to write a column such as this. To accuse my fellow Muslims of being soft on the scourge of anti-Semitism isn’t easy; I feel as if I am “dobbing in” the community, telling tales to the non-Muslim teacher. Nor do I particularly want to assist the English Defence League in its relentless campaign to demonise all Muslims, everywhere, as extremists and bigots.

We aren’t. And we’re not all anti-Semites. But, as a community, we do have a “Jewish problem”. There is no point pretending otherwise. That Bradford’s Council for Mosques has been campaigning to save the city’s last remaining synagogue from closure doesn’t change the fact that thousands of British Muslims will have been nodding in agreement as they read Lord Ahmed’s comments about Jewish power and influence – or will have assumed that the Times scoop is evidence in its own right of a “Zionist plot” against the peer. Oh, and I’m well aware that this column will also be held up by some of my fellow Muslims as “proof” that “Mehdi Hasan has sold out to the Jews”.

I only hope and pray that Lord Ahmed’s comments will act as a wake-up call to Britain’s moderate Muslim majority. The time has come for us to own up to a rather shameful fact: Muslims are not only the victims of racial and religious prejudice but purveyors of it, too.

In 2011 Baroness Warsi, the then Conservative Party chairman, said that Islamophobia has “passed the dinner-table test” in polite British society. I agree with her, but what she omitted to mention, and what we Muslims must now admit, is that anti-Semitism passed the dinner-table test in polite British Muslim society long ago.

Mehdi Hasan is a contributing writer for the New Statesman and the political director of the Huffington Post UK, where this column is crossposted