Last week’s Heritage Foundation panel on the 2012 attacks in Benghazi was bound to be an ugly affair, what with the presence of panelist Brigitte Gabriel, a self-described “terrorism analyst” with a laundry list of offensive statements about Islam and Arabs. Sure enough, when attendee Saba Ahmed, an American University law school student, explained that not all Muslims are terrorists, Gabriel retorted that “the peaceful majority were irrelevant” in the Sept. 11, 2001, attacks in the way that peaceful Germans were irrelevant during the Holocaust.

That prompted much hand-wringing, primarily on cable news, about the supposed silence of “moderate Muslims” in this supposed age of Islamist extremism. What no one on either side of the debate questioned, though, was the legitimacy of the phrase "moderate Muslims" itself.

In the years that I’ve spent writing about and studying the phenomenon of Islamophobia, that phrase has always troubled me. Muslims and non-Muslims alike bandy it about, though the latter usually demand that the former prove that they are such. What bothers me is not that there aren’t “moderate Muslims”—from my perspective, there certainly are. The unacknowledged problem is how that phrase informs our judgments.

Brigitte Gabriel, for instance, told the Australian Jewish News in 2008, “Every practicing Muslim is a radical Muslim,” meaning “moderates” must be only those who don’t practice their religion. Celebrity atheist Sam Harris writes that “moderate Muslims” are those who express skepticism over the divine origins of the Quran and “surely realize that all [sacred] books are now candidates for flushing down the toilet.” Then there’s conservative columnist John Hawkins, who enumerates seven criteria that Muslims must meet in order to be considered “moderate” while the queen of Muslim-bashing, Pamela Geller, asks in typical fashion, “What’s the difference? Today’s moderate is tomorrow’s mass murderer.”

To be fair, it’s not just the wackos. Newsweek, NPR, the Wall Street Journal, Reuters, TIME, The New Republic and many others have used this phrase to describe Muslims who fit a certain preferred profile. Many Muslims themselves have bought into this dichotomy, if only to distance themselves from the so-called radicals and extremists—to assure paranoid non-Muslims, in other words, "I'm not that kind of Muslim."