As Marge Schott-impersonator Donald Sterling and honorary professor of Afro-American Studies Cliven Bundy could tell you, nothing writes you out of polite society more quickly than being outed as a racist.

Which may be one of the reasons that politically savvy Democrats are never slow to equate advocates of limiting the size, scope, and spending of the federal government with racism, slavery, and white supremacy. Who can blame them, really? Even after the “success” of Obamacare, the president somehow has managed to chalk up his lowest approval ratings ever, and things don’t look so good for the Donkey Party in the fall’s midterm elections, either.

Salon’s Joan Walsh is quick to cry racism in the face of arguments or developments she doesn’t like, as are MSNBC hosts Chris Matthews, Lawrence O’Donnell, and Ed Schultz. Jimmy Carter, who himself stooped to race-baiting during his 1970 campaign for governor of Georgia, has chalked up “an overwhelming portion” of negativity toward Barack Obama to the fact that “he is a black man.”

Then there’s New York magazine’s Jonathan Chait, who recently averred that “America’s unique brand of ideological anti-statism is historically inseparable…from the legacy of slavery.” Unlike many of his liberal-progressive confrères, Chait recognizes that “advocating tax cuts is not in any meaningful sense racist” and that even if small-government conservatives and libertarians were really secretly bent on moving the nation’s capital to Stone Mountain, that doesn’t mean particular policy proposals can simply be written off: “Individual arguments need and deserve to be assessed on their own terms, not as the visible tip of a submerged agenda; ideas can’t be defined solely by their past associations and uses.”

There’s no question that some of the past associations are ugly. To the extent that many Republicans, conservatives, and libertarians look back kindly on Barry Goldwater’s 1964 shellacking at the hands of Lyndon Johnson, they have to acknowledge that opportunistic segregationists came along for the ride (as did a young Hillary Clinton, who was hardly a racist). As Glenn Garvin writes: “Nothing was more problematic than the civil rights issue—particularly the 1964 Civil Rights Act, which outlawed most forms of racial discrimination. Goldwater was no racist; early in his career as a Phoenix city council member, he aggressively supported local civil rights ordinances…Goldwater was privately appalled to discover that his opposition to the Civil Rights Act rallied to his side not only libertarians but racists who detested and feared not state power but black people. He was horrified when Alabama’s racist Gov. George Wallace offered to switch parties and run as his vice president.”

Characters such as Wallace and South Carolina Sen. Strom Thurmond, a longtime Democrat who migrated to the Republican Party in the civil rights era, were hardly limited-government types. While decrying virtually all federal spending not directly tied to defense, Thurmond—certainly one of the most disgusting American politicians of the second half of the 20th century—also pushed pork-barrel projects and protectionist politics for the Palmetto State, its textile industry, and even historically black colleges. You can’t be a segregationist and believe in a minimal government, really: Policing the color line is an exhausting job that requires a vast, expensive apparatus.

To Goldwater’s discredit, however, he didn’t confront the contradictions among his followers. As left-wing biographer Rick Perlstein grants, Goldwater was a man of color-blind temperament, conviction, and personal action. His family integrated its department store long before it was common, and he founded the Arizona Air National Guard “as an integrated unit.”) But it’s equally clear that “Mr. Conservative”’s statement of principles wasn’t fully up to addressing the challenges of a still-segregated America: “Our aim, as I understand it, is neither to establish a segregated society nor to establish an integrated society,” Goldwater said. “It is to preserve a free society.” Had Goldwater followed that sort of statement with full-throated invocations of a truly inclusive America, he might have garnered even fewer votes than he managed against LBJ. But he also would have helped to keep calls for a smaller federal government from being seen as a backdoor attempt at Jim Crow.

But contra William Faulkner, there are signs that the past is finally becoming past. Certainly there’s no credible way to mistake the contemporary libertarian agenda for the second coming of Thurmond. That’s true even after the 2008 revelation by James Kirchick of newsletters published under Ron Paul’s name in the 1980s and ’90s that were filled with racist and homophobic material. Paul, who served decades in Congress as a libertarian-leaning Republican and ran for president in 1988 on the Libertarian Party ticket, is without question the politician most responsible for the boom in limited-government and libertarian rhetoric. There is no defending the risible publications (whose authors Paul has refused to identify), which claimed, among other things, that Martin Luther King Jr. “seduced underage girls and boys” and that AIDS sufferers “enjoy the attention and pity that comes with being sick.”

It’s equally clear that the reason Paul topped The New York Times’ best-seller list; pulled plaudits from Andrew Sullivan, Jake Tapper, George Will, Vince Vaughn, and others; and packed college campuses had absolutely nothing to do with such idiotic and offensive statements, which even most movement libertarians knew nothing about until 2008. Rather, his appeal proceeded (and still proceeds) from a radical message of individual autonomy and decentralized political power. In 2008, my Reason colleague Brian Doherty noted, Paul wrapped up his typical stump speech with: “I don’t want to run your life. We all have different values. I wouldn’t know how to do it, I don’t have the authority under the Constitution, and I don’t have the moral right...I don’t want to run the economy. People run the economy in a free society...“I don’t want to run the world…We don’t need to be imposing ourselves around the world.”

The fixations of small “l” libertarians include ending the drug war, mandatory minimum sentence and other prison reforms, and pushing a maximalist version of school choice, all of which would directly benefit minorities more than non-minorities. Libertarian public-interest law firms such as the Institute for Justice spend much of their time fighting occupational licensing laws that disproportionately stymie inner-city entrepreneurs who have little to no political or economic capital. IJ’s first case, dating back to 1991, attacked Washington, D.C.’s absurd laws against African hair-braiding without expensive and irrelevant cosmetology licenses.

Similarly, there’s no way to confuse libertarian obsessions with Fourth Amendment rights, ending stop-and-frisk policies, and reversing “the rise of warrior cops” with anything related to white supremacy. The same goes for the libertarian insistence against an interventionist foreign policy, whether through boots on the ground or via drone strikes and bombing runs. As with any group, there are differences, but libertarians have long been in the forefront of pushing for legalized abortion and gay marriage. (Reason magazine, like the Libertarian Party, was calling for the legalization of same-sex marriage in the early 1970s, when the Diagnostic and Statistical Manual of Mental Disorders still considered homosexuality a form of mental illness that should be “cured.”)

For libertarians, these things are all of a piece: Limit the size, scope, and spending of government at all levels, and increase the ability of individuals and groups to run as many “experiments in living” (as John Stuart Mill would say) as possible. An increasing number of Americans are grokking libertarian fears about growing federal power, with a record-high 72 percent agreeing that “big government” represents a bigger threat to the future of the country than “big labor” (5 percent) or “big business” (21 percent).

Many—maybe most—Americans hoped that the election (and reelection) of Obama would put an end to racial division and enmity in a country that has never lived up to its stated promises of equality and individual rights. That surely hasn’t happened. But if the broad-based reactions to Donald Sterling and Cliven Bundy tell us anything, it’s that these old men—combined, they are over 140 years old—remind us of where America was, not where it is today. And so too do efforts by one of the GOP’s leading presidential candidates, Sen. Rand Paul (R-Ky.) to reach out to black and minority audiences. Paul, the son of Ron who has had his own problems related to race, is leading the charge to end mandatory minimums, do sentencing reforms, restore voting rights for felons, and more.

In a political context, appeals to racial solidarity and racial division will never disappear, but they will become less and less meaningful, especially in an America that is cohering around the idea that we’re all in this together—against a government that threatens us all.