Barack Obama went too far Monday in his Rose Garden comments about the Supreme Court. He also walked those comments back pretty clumsily on Tuesday. But let’s not lose sight of the big things here. Obama’s lousy word choices aside, the heart of his criticism is right, as conservatives like that grandstanding judge on the Fifth Circuit surely know. This controversy is about the threat of a massive case of judicial activism from the people who say they loathe judicial activism, and if the Supreme Court repeals the individual mandate—or especially the whole Affordable Care Act (ACA)—the court will be seen as, and will be, just another wheeling-dealing political institution, no different from Congress or the Chicago Board of Aldermen. And that, in turn, will open the court up to even more politicization by partisans and activists. John Roberts and Anthony Kennedy may disagree with the mandate, but they really want to think this through.

Obama’s error in his Monday comments concerned the phrase “an unelected group of people.” The Supreme Court may be not trusted by three quarters of the American public anymore, but they aren’t just nine people off the street, or nine generals pulling a Seven Days in May on us. They’re Supreme Court justices. They got where they got through the proper constitutional procedures. Like them or not, following Marbury v. Madison (which established the precedent of judicial review) is their right as much as it was William Brennan’s. I can well imagine that if George W. Bush had suggested that a liberal-majority Supreme Court was just “an unelected group of people,” liberals would have howled.

Obama did better on Tuesday, but he still got some things wrong. I think he was using the word “Lochner” as shorthand for “the Lochner era,” the era of conservative court dominance from 1905 (when Lochner v. New York was decided) to 1937. It’s a shorthand liberals often use (meaning, really, “Lochner-era jurisprudence”). But he also wasn’t right that the court has never overturned Congress on an economic issue. The most notable of those cases were right at the end of the Lochner era. If the president is going to make remarks the White House knows are going to generate controversy, he’d better choose his words very carefully and get all his facts exactly right.

But Obama’s main point was spot on. Conservatives have for years preached judicial restraint, arguing that the court’s default position should be to defer to Congress except in cases where a law is very obviously unconstitutional. That is hardly the case here. Whatever our disagreements about the ACA, even many conservatives admit it’s a tough, close call, and some leading conservative judges have ruled against the challenges.

But of course, and as with everything these alleged moral absolutists do, their argument about restraint was contingent on circumstances. Conservatives started talking up restraint back in the early 1980s, when the court had a strong liberal majority that conservatives wanted to stop from being activist. And they haven’t minded saying it in more recent years, as Republican control (often enough, anyway) of at least one house of Congress ensured that the legislative branch wouldn’t do anything too liberal in the first place that the court would have to overturn.

But now, in this one instance, the poles are reversed. Now, a liberal(ish; let’s not go overboard) Congress passed a law while a conservative majority sits on the court. Ach, cry conservatives—now is no time for restraint! Give me a break.

John Roberts has to know and see all this. He has to know that Fifth Circuit Judge Jerry Smith, who asked federal prosecutors for a homework assignment in the wake of Obama’s remarks—a brief stating the Justice Department’s position on judicial review, that had to be at least three pages, single-spaced!—is making conservatives look silly and cheapening the bench. And he has to know that the court’s reputation will suffer an immense blow if it overturns the mandate. It will be seen by a large majority—even a lot of people who weren’t crazy about the law—as completely political. Remember, they didn’t have to take the case in an election year in the first place. They could have put it off. But the court said it must do this now. If it then overturns the ACA, it will look and smell like a political hit job to many Americans. And the court would be saying to America, “We know what you think, and we don’t give a damn.”

What would happen to the court then? Slowly—no; probably quickly—it will come to be seen by most Americans as just another cesspool of political mud wrestling; just another arena where the rich get what they want while everyone else gets screwed (Citizens United); just one more ideological whorehouse full of patrons pretending to be just the piano player.

Despite what we’re all brought up to believe, nothing about the court is sacrosanct. Lifetime appointments can be changed to fixed-year terms. It’d take some doing, but it can be done. And there’s nothing anywhere that says it has to be nine justices. That’s just tradition, but it’s nowhere in the Constitution. It just needs to be an odd number; could be three or 23. For that matter, Congress could disregard Marbury v. Madison. Yep. It could. Tom DeLay used to speak of this from time to time, back in the dear old Terri Schiavo days. He never specifically invoked M v. M, but, referring to judges who would have let Schiavo die, he said things like they had “thumbed their noses at Congress and the president” and would someday pay. He meant a campaign against judicial review. He never got around to it, having been indicted and convicted and all, but that’s what he meant. There’s nothing to prevent liberals from mounting a similar campaign. So far they’ve has held back by their respect for the institution. But that may soon be gone.

The court already—after Bush v. Gore and Citizens United in particular—has one leg dangling off the cliff. If it is going to be nakedly political in its decisions, it will invite political blowback. And if it gets pushed off the cliff, then who or what is our final arbiter in America? The great irony of the conservative moral absolutists is that they have ushered in—by arguing that nothing is above or beyond politics, that all the institutions Americans used to respect were in fact infested with liberal biases and presumptions—the greatest period of relativism in American history. We no longer universally respect anything. There is no reason the court should be immune from that. If the ACA is gutted, it will be liberals who will seek revenge on the court, but remember that it was conservatives who opened the door to pure politicization in the first place.