The right has always been against race-conscious remedies to racial discrimination, touting “colorblindness” as the “constitutional” approach to making policy. But it’s only been in the last five years—since the election of Barack Obama—that it’s scored significant victories.

Led by figures like Hans von Spakovsky—a former Bush official who once accused the Obama Justice Department of anti-white racism—and boosted by conservatives on the Supreme Court, the right has chipped away at affirmative action and the Voting Rights Act, while using their state-level clout to limit voting with strict ID requirements and attacks on early and weekend voting.

But there’s still more to do. Von Spakovsky—and his frequent collaborator, Roger Clegg of the misnamed Center for Equal Opportunity—have gone after congressional remedies to the court ruling, calling it a “ ‘get out of jail free’ card to black elected officials in the South, where they can discriminate all they want against white voters.” They are prepping an assault on Section 2 of the VRA, which bars actions that cause racial discrimination in voting, even if that’s not the intent. They’ve also opened a new front in the war against civil rights laws, with an attack on the Civil Rights Act itself.

“Discrimination on the basis of race and ethnicity is unconstitutional, unlawful, and morally repugnant, yet the practice is rife throughout federal law and government programs,” wrote Clegg, von Spakovsky, and Elizabeth Slattery in National Review this week. What they mean, of course, are the laws and regulations designed to prevent and ameliorate the effects of racial bias in hiring, education, voting, and other areas. To use a quote from Chief Justice John Roberts, a fellow traveler in the fight to end race-consciousness, “The way to stop discrimination on the basis of race is to stop discriminating on the basis of race.”

Von Spakovsky, Clegg, and Slattery focus their fire on the doctrine of “disparate impact,” which treats neutral actions with racially disproportionate outcomes as illegal, if—for example—a business or institution can’t justify a practice as necessary to the job. For them, “Eliminating such claims is therefore another way to help curb the use of racial and ethnic preferences.”

But this is ahistorical nonsense. “Disparate impact” exists because discrimination was often achieved by neutral means.

During Jim Crow, for instance, explicitly discriminatory voting was illegal. White Southerners could block blacks from using public facilities or mandate segregated businesses, but they couldn’t bar blacks from voting. Hence the poll tax and the literacy test. In theory, they were universal requirements—everyone was vulnerable to failing the test or lacking the funds to pay a tax. In practice, of course, extreme poverty and deprivation meant that ex-slaves and their descendants were most likely to fail the test or lack the funds. The same went for felon disenfranchisement; in theory, everyone who committed the felony of vagrancy or theft could lose his or her voting rights. In practice, however, these crimes were selectively applied to blacks.

It’s for this reason that lawmakers built disparate impact into the 1964 Civil Rights Act (Title VII) and the Age Discrimination in Employment Act of 1967. Even today, disparate impact is critical to fairness in employment, housing, and other fields. The Obama administration has used disparate impact claims to win settlements from banks accused of predatory lending toward minorities. The simple fact is that racial bias is still alive in vast areas of American life, and most people who discriminate are too smart to broadcast their prejudice.

To von Spakovsky, Clegg, and Slattery, however, these measures are as bad, if not worse, than racism against minorities. It doesn’t matter that racial inequities move from generation to generation, propelled by the force of past and present discrimination; for these conservatives, any acknowledgment of race is racism, and anti-racist policies—like federal civil rights laws—are a zero-sum game, with whites as the losers.

I mentioned at the beginning that the right has always been against race-conscious laws. That’s understating the case. Movement conservatives are the most prominent proponents of the “colorblind” approach to law and policy, but they merely picked up the baton from previous opponents of race-conscious policy.

Take President Andrew Johnson, who vetoed the 1866 civil rights bill with a message that echoes von Spakovsky’s view that the “real racists” are those who take race into consideration. “In all our experience as a people living under Federal and State law,” writes the unpopular (and soon-to-be impeached) president, “no such system as that contemplated by the details of this bill has ever before been proposed or adopted. They establish for the security of the colored race safeguards which go indefinitely beyond any that the General Government has ever provided for the white race.” In fact, he continues, “the distinction of race and color is by the bill made to operate in favor of the colored against the white race.”

This was borrowed later by Justice Joseph P. Bradley in his ruling against the Civil Rights Act of 1875, which he struck down as unconstitutional. Bradley wrote:

When a man has emerged from slavery, and by the aid of beneficent legislation has shaken off the inseparable concomitants of that state, there must be some stage in the progress of his elevation when he takes the rank of a mere citizen, and ceases to be the special favorite of the laws, and when his rights as a citizen, or a man, are to be protected in the ordinary modes by which other men’s rights are protected.

Circumstances change and ideologies shift, but the message from conservatives stays the same: What happens on the ground doesn’t matter; equality under the law is sufficient for civil rights. This has never been (and isn’t) true, but then, this isn’t a matter of truth as much as it’s a question of belief. If you believe that racism has been sorted away—or if it hasn’t, that it isn’t a concern for the federal government—then there’s no need for active efforts toward racial amelioration. And if, like von Spakovsky, you see anti-white discrimination as the key concern, you’ll oppose any effort to level the playing field for minorities.

But if you see racism as a force to fight—if, in other words, you think the facts matter—then you’ll reject this “colorblindness” for what it is: a reactionary excuse for doing nothing.