Justice Antonin Scalia quite deservedly came under fire yesterday for his claim that a key provision of the Voting Rights Act is a “perpetuation of racial entitlement.” If the justice were looking to confirm every suspicion that conservative opposition to the law that broke the back of Jim Crow voter exclusions is rooted in white racial resentment, he could hardly have picked a better way to do so.

Viewed in context, however, Scalia’s quote is actually even more disturbing than the initial headlines suggested. Beyond whatever resentments Justice Scalia may hold, his “racial entitlements” statement was also part of a broader theory about the proper role of judges in society. And if that theory were taken seriously by a majority of the justices, it would potentially undermine Medicare, Social Security and countless other programs. According to Scalia:

Whenever a society adopts racial entitlements, it is very difficult to get out of them through the normal political processes. I don’t think there is anything to be gained by any Senator to vote against continuation of this act. And I am fairly confident it will be reenacted in perpetuity unless — unless a court can say it does not comport with the Constitution. You have to show, when you are treating different States differently, that there’s a good reason for it. That’s the — that’s the concern that those of us who — who have some questions about this statute have. It’s — it’s a concern that this is not the kind of a question you can leave to Congress. There are certain districts in the House that are black districts by law just about now.And even the Virginia Senators, they have no interest in voting against this. The State government is not their government, and they are going to lose — they are going to lose votes if they do not reenact the Voting Rights Act.

As Scott Lemieux points out, this theory resembles some of the reasoning behind an 1883 decision which struck down an early precursor to the 1964 Civil Rights Act that banned many forms of segregation by private business. But the roots of Scalia’s legal theory are probably several decades more recent than the late Nineteenth Century.

A major reason why U.S. constitutional law is difficult and American judges are powerful is that the Constitution is not a particularly precise document. The Ninth Amendment provides that “[t]he enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others retained by the people,” but it provides absolutely no guidance whatsoever on what those other rights could be. The Fourteenth Amendment prohibits states from abridging the “privileges or immunities of citizens of the United States” and it prevents the loss of life, liberty or property “without due process of law.” What are the “privileges or immunities” protected by the Constitution? The Constitution doesn’t say! And the Supreme Court’s answer to this question is rather ridiculous.

Because the Constitution is so imprecisely drafted, judges have at times wielded it as a tool to impose their own policy preferences on the law. After all, nothing in the Constitution doesn’t say that the word “liberty” means the liberty to work in a grueling job that literally tears your lungs apart from the inside without the benefit of workplace safety laws or other basic labor protections. Indeed, that was more or less the holding of the Supreme Court’s 1905 opinion in Lochner v. New York.

Beginning in the 1930s, the Supreme Court more or less came to terms with the fact that judges were fabricating imaginary constitutional restrictions, and that this was a bad thing. Indeed, probably the most important question addressed by Twentieth Century constitutional law and scholarship was how to reconcile the legitimate need for judges to police the boundaries of liberty with the fact that the Constitution provides them with little guidance on how to do so. An unelected Supreme Court had recently engaged in overreach that harmed the most vulnerable Americans and struck down laws enacted by elected representatives in the process, and this situation was untenable. But a Court that simply gave Congress free reign to engage in racism or sexism or to toss people in jail for no reason would be equally intolerable.

One of the best, and most influential answers to this dilemma came in a 1938 case called United States v. Carolene Products. Carolene Products suggested that judges should step into the breach when a law “restricts those political processes which can ordinarily be expected to bring about repeal of undesirable legislation.” Thus, when lawmakers enact “restrictions upon the right to vote,” limit the ready availability of information to voters, or restrict political organizing, judges must remedy the situation. Carolene Products also provides that judges must enforce the Bill of Rights and other explicit constitutional protections, and that it must prevent discrimination against certain minority groups, but in most circumstances we are governed by democratically elected representatives and judges should defer to officials who can actually be voted out of office.

When Scalia uses the term “racial entitlement” he appears to be referring to the kind of law that entrenches itself because lawmakers are too afraid to vote against it for fear of being accused of racially improper motives. As Scalia puts it, “[w]henever a society adopts racial entitlements, it is very difficult to get out of them through the normal political processes.” In other words, Scalia believes that the Voting Rights Act somehow “restricts those political processes which can ordinarily be expected to bring about repeal of undesirable legislation,” and thus it is his job as a judge to strike it down.

This is a disturbing idea for many reasons, but one of the biggest ones is that its logic could extend well beyond the Voting Rights Act. There is a common belief among conservatives that welfare programs by their very nature lead to the kind of so-called breakdown of democracy that Scalia finds objectionable in the Voting Rights Act case. Indeed, the most famous articulation of this view was Mitt Romney’s 47 percent remark: “those that are dependent on government and those that think government’s job is to redistribute — I’m not going to get them.” In essence, Romney warned that as the government creates welfare programs, this transforms welfare recipients into a constituency for those programs. And eventually that constituency becomes so large that it is impossible for a lawmaker to repeal those programs, or for people who oppose those programs to get elected.

To be sure, Scalia has never explicitly endorsed Romney’s view of welfare — although I’d be willing to make a $10,000 bet that he agrees with Romney. But it’s not hard to predict how a judge who agrees with both Romney’s view of welfare and Scalia’s view of when judges must destroy democracy in order to save it would react to the modern welfare state. With his racial entitlement comment, Scalia offered a constitutional theory that would allow movement conservatives to strike down the entire American safety net.

The irony in all this is that Scalia used to be one of the most articulate spokespeople for why judges should not strike down laws unless the Constitution’s text explicitly instructs them to do so. Just last year he denounced Roe v. Wade specifically because he does not believe that supporters of Roe “stick to the text” of the Constitution. Whatever the virtues of Roe, Scalia appears perfectly willing to abandon his rigid textualism when it comes time to strike down a law he personally disapproves of. I’ve read the Constitution many times, but I’ve never found a ban on what Scalia calls “racial entitlements.”