More than a few on the right are alarmed by the torrent of racism accusations that an all-star cast of liberal luminaries has directed at President Obama’s critics. “When people like Jimmy Carter and Maureen Dowd start saying this kind of thing, we’ve reached a whole new level of ugliness in our political discourse,” as one friend of mine puts it. “No charge in American life is so poisonous.” He’s got a point, of course—especially about the poisonous part. Yet as those on the other side spew the R-word with ever more irresponsible abandon, some of us find new reason for hope. Here’s a chance to lance the boil once and for all.

Genuine racism is a terrible thing, and for far too long it was a virulent strain running through our national life. This is so patently obvious that it scarcely bears repeating. Yet those of us who point out how much our nation has changed for the better invariably feel obliged to repeat it, early and often, lest our very sense of optimism about race relations make us subject to the charge. So while we’re at it, let’s dispense with the other essential pro forma acknowledgment: yes, even in today’s America, traces of the vilest racism persist in some dark hearts and twisted minds.

But those liberals who’ve lately been issuing the racism charge so promiscuously (speaking of aberrant hearts and minds) are aiming it not at skinheads living in their parents’ basements or at would-be Klansmen, but at decent Americans with the temerity to object to presidential policies that they believe would damage both the quality of their lives and the nation itself: in short, at Americans acting in the best tradition of democratic citizenship. This is so preposterous that literally millions who’ve never before given the matter any thought are taking notice.

And what they see is what has long been true: that the charge of racism is invariably a crock; indeed, that more than simply an expression of (often contrived) liberal moral outrage, it’s intended to be the ultimate conversation stopper. Ward Connerly, long the leader of the fight against racial preferences in America, observes that he’s had the experience more times than he cares to count of speaking before an audience, knowing that 99 of 100 people agree with him. “But if there’s one angry black person in the audience who disagrees,” he says, “that person controls the room. He’ll go on about the last 400 years, and institutional racism, and ‘driving while black,’ and the other 99 will just sit there and fold like a cheap accordion.” And Connerly is black himself. For the liberal opinion makers and trend setters who’ve set themselves up as America’s racial referees, the accused racist is always presumed to be guilty of at least something.

I speak here from all-too-personal experience. Some years ago, promoting a book about my move from left to right, I gave a speech in Dallas in which I chanced to mention the ongoing controversy over Mark Twain’s use of the N-word in Huckleberry Finn. Afterward, in the question-and-answer segment, an audience member took exception to my own use of the offending word (in quoting Twain), and we had a brief back-and-forth about it. I figured that was the end of it, until the Fort Worth Star-Telegram ran a hit piece on the “conservative author” who’d supposedly made racially inflammatory comments in a speech, thereby leaving my audience “stunned.” In short order, several other Texas papers picked up the story.

So there I was, publicly labeled a racist. It didn’t feel terrific. For a few days there, I wanted nothing so much as to hide under the nearest rock. That’s why I will be forever grateful to Myron Magnet, then the editor of City Journal, who persuaded me to write about the incident for the magazine instead. The result was cathartic. Readers reacted with anger and disgust not only on my behalf but also, in many cases, because they clearly identified, having been subject to the baseless charge also, if only in social situations. They deluged the offending reporter and his editor with e-mail; some who lived in the Dallas area canceled their subscriptions to the Star-Telegram. For once on the defensive, their dirty work exposed to public view, the race baiters could only issue mealy-mouthed self-justifications.

Much the same thing is beginning to happen today on a much vaster scale. Countless ordinary Americans are tired of being called racists for no other reason than that their political and social beliefs are uncongenial to those on America’s college campuses or New York’s Upper West Side. President Obama himself may have set this seismic shift in motion in July with his off-the-cuff and wholly inaccurate attribution of racism to a decent Cambridge cop for the sin of doing his job. But the latest spate of charges from his surrogates is the topper. Here’s hoping it leads to the honest conversation on race that Obama’s attorney general, Eric Holder, claims to want.

That conversation is long overdue, so let’s have at it. Let’s talk, for starters, about the shocking double standard in the way liberals and conservatives are allowed to deal with race and racism. Why is it okay for liberals to belittle Clarence Thomas endlessly as an Uncle Tom? And how does liberal cartoonist Ted Rall get away with calling Condoleezza Rice a “house nigga,” and his colleague Jeff Danziger with drawing her as a mammy in a caricature as cruelly demeaning as anything in Julius Streicher’s Der Stürmer?

Let’s talk, too, about racial profiling—and start paying appropriate attention to the statistical evidence cited by Heather Mac Donald establishing that the disproportionate arrest and incarceration rates of minorities are explained not by racism but by disproportionate rates of criminality. Let’s talk about how American business has long been subject to blackmail by Jesse Jackson and Al Sharpton in the name of social justice, and about the many other ways in which the regime of racial preferences has sowed division and corruption in this country. Let’s talk about how even after the Duke rape fiasco, the media continue to give credence to every racism charge; indeed, how just this week, vicious (and transparently phony) statements about race attributed to Rush Limbaugh uncritically disseminated by mainstream outlets helped sink Limbaugh’s bid for NFL ownership. And yes, let’s talk about white liberal bigotry, the bigotry of low expectations, and how it cripples and demeans those it supposedly aims to help. Exhibit A might be the recent call by the Tucson Unified School District to revamp its disciplinary system to cut down on the suspensions and expulsions of minority students (but not white ones) so that the numbers reveal “no ethnic/racial disparities.”

Are such conversations possible in contemporary America? With the liberals’ racism charge losing its power to intimidate and silence, there is at least some hope. Because finally, more and more of us are getting the message that it’s the fear of having these conversations that is truly racist.