Ian Millhiser of ThinkProgress and political scientist Brendan Nyhan recently had a fascinating discussion on Twitter—a phrase I don’t often use—about the condition of American democracy (not good, all agreed) and what that meant for the next time Democrats gained control of both Congress and the White House. Even with a unified government, anti-democratic features in the American system, including a Supreme Court that leans right thanks to a stolen seat, will make genuine progress difficult to impossible. So what should Democrats do? The discussion essentially concerned what the legal scholar Mark Tushnet labeled “constitutional hardball”: that is, the use of legal but non-normative means to pursue positive ends. When it comes to the Supreme Court, Millhiser wrote, Democrats “may need to make some very difficult choices”, including whether to pack the Court.

While the judiciary in America has accrued a substantial amount of power, this power is much more fragile than is commonly assumed. Article III of the Constitution gives Congress the power to establish (or not) lower courts as it sees fit, and also to expand the size of the Supreme Court. Congress can also make “exceptions” and “regulations” to the appellate jurisdiction of the courts. These potentially powerful tools make the judiciary vulnerable to constitutional hardball.

Why has judicial power paradoxically thrived given the weapons that Congress can theoretically use to cut it down to size? Quite simply, because political élites generally favor it. This shouldn’t be all that surprising. Supreme Court justices, after all, are nominated by presidents and confirmed by Senate majorities, and despite the myth of the courts as a “counter-majoritarian institution,” the Supreme Court is rarely far outside the political mainstream. In some cases, like the conservative courts of the Gilded Age or the Warren Court at the height of liberalism under LBJ, the courts are active partners of a dominant national governing coalition.

Since early in the Nixon administration, the median vote on the Court has been a moderate Republican, such as Lewis Powell, Sandra Day O’Connor, and now Anthony Kennedy. These courts have generally been conservative, but deliver enough victories to both sides to maintain elite and popular support for the institution. Indeed, in part because liberal victories under the Roberts Court have been fewer but tend to be higher-profile (most notably on same-sex marriage, abortion rights, and two cases involving the Affordable Care Act), the Court is actually more popular among liberals than conservatives.

But it’s crucial to remember that this equilibrium is contingent, not inevitable. Republicans during Reconstruction manipulated the size of the Court, briefly adding a tenth member, and while FDR’s initial Court-packing proposal—which was presented in an uncharacteristically ham-handed manner—failed, the constitutional crisis that compelled it quickly faded as Justice Owen Roberts started voting with the Court’s liberals to uphold New Deal programs. Soon after, retirements allowed FDR to make enough nominees to ensure a Court that would not interfere with the core New Deal agenda.