Shortly after the conclusion of the 2015 term of the United States Supreme Court, Justice Ruth Bader Ginsburg told the New York Times: “I can’t imagine what this place [the Supreme Court] would be . . . with Donald Trump as our president.” In a subsequent interview she called Trump a “faker” who “says whatever comes into his head at the moment.” She later acknowledged that her remarks were ill-advised and she promised to refrain from such political partisanship in the future. Oddly, her break with the longstanding judicial code of nonpartisanship was, in a way, Trumpian. She said out loud what even casual court-watchers seldom acknowledge, but know to be true. On many issues of contemporary political consequence, it is the men and women of the courts, not the law, who rule.

A core premise of American constitutional government is that we live under the rule of law, not the rule of men. But we are very far from that ideal today. With the death of Antonin Scalia, the political balance of power on the Supreme Court may shift after the next presidential administration. If Hillary Clinton is elected, Scalia’s replacement will almost certainly vote with the liberal Justices, meaning that certain constitutional provisions will be interpreted in a new way. But how can it be that the death of a single Justice will result in different binding interpretations of the laws? Isn’t that the definition of the rule of man?

The cynic or the realist might suggest that the foregoing questions are naive. Of course politics play a role in Supreme Court decision-making. We have known that since the Federalists managed to secure control of the courts over two centuries ago. But still, there is a troubling contemporary acceptance of a politicized judiciary. There is little discussion of what might be done to make the courts less partisan and much concern by both major parties that future judges demonstrate partisan bona fides. Among a significant number of Republican voters, support for Donald Trump in the upcoming election is founded entirely on the expectation that he will appoint conservative federal judges (and that Hillary Clinton would appoint liberal ones).

On particular issues, how individual Justices will rule is fairly predictable. Democrat appointees will rule in favor of abortion rights, gay rights, public employee rights, and affirmative action, and against property and other economic rights claims. Republican appointees will rule in the opposite direction. Of course, there have been exceptions, like Earl Warren and David Souter, but once they revealed their true colors we could rely on how they would rule. And there have been the so-called swing Justices like Anthony Kennedy, who keeps us guessing, and occasional deviations by Justices thought to be reliably liberal or conservative. For example, Republican appointee Justice Sandra Day O’Connor ruled in favor of abortion rights in Planned Parenthood v. Casey, though she would have been with the dissent in Roe v. Wade. But there can be little argument that on questions like those noted above, the Justices (and most federal judges) usually find that the law means what those in the political party of the President who appointed them would want it to mean.

A handful of cases from the Supreme Court’s most recent term, all decided after Justice Scalia’s death, illustrate the point. Although we cannot know for certain how Justice Scalia would have voted, there is little doubt that as many as five cases would have been decided differently with Scalia still on the Court. The Court issued per curiam (unsigned) opinions in five cases in which the Justices were divided 4-4. Scalia would have broken the tie in every one, likely leading to different outcomes in three of the five. Those cases include Friedrichs v. California Teachers Association (upholding mandatory public union dues), Geneva College v. Burwell (upholding the individual mandate under Obamacare), and Dollar General v. Mississippi Band of Choctaw Indians (upholding tribal jurisdiction over non-tribe members). In a fourth case, Fisher v. University of Texas, Scalia’s presence would almost certainly have led to an affirmance, rather than an overruling, of the lower court’s opinion invalidating the university’s affirmative action program. Since the conclusion of the 2015-2016 term, the Court issued another ruling declining to stay a Fifth Circuit invalidation of North Carolina voting laws, a case that would likely have had a different result were Scalia still alive.

On the nine-member Court, it is a given that a single Justice’s vote will make a difference when the other Justices are evenly divided. But it is one thing to be divided over questions of legal interpretation, and something else entirely to be divided over philosophical or policy questions. The latter is easily foreseen based on the politics of the appointing President, while the former has little, if anything, to do with philosophical or policy preferences. Disagreements over legal interpretation are difficult to predict without an understanding of each Justice’s previous interpretations. Differences over policy are easily predicted even if we know little or nothing about the law.

Yet one would be hard pressed to find a judge who admits to partisanship in his or her rulings. And with respect to most federal court cases, judicial claims of nonpartisanship are justified, including in the Supreme Court. The court issued 80 decisions in its last term, 36 of which were decided unanimously. In 29 of the cases in which the Court was divided, the majority and/or the dissent included both liberal and conservative justices. Justice Clarence Thomas and Justice Ginsburg agreed 62 percent of the time, the lowest level of agreement for any two members of the Court. Chief Justice John Roberts agreed with Justice Stephen Breyer 84 percent of the time. So it’s not that politics rules everything.

But politics does appear to rule in most of the cases that draw public attention. It might be argued that those cases draw such attention because the Court has split along apparently political lines, but the reality is that they draw it long before they are decided—with the left and the right expecting that “their” Justices will rule along party lines. And there is predictable outrage when a Justice deviates from what is expected. Clearly, the public sees the Supreme Court as a political institution and that is not good for the rule of law or an independent judiciary.

The rule of law depends as much upon public confidence that judges will be nonpartisan as it does upon judges actually being nonpartisan. Realists might suggest that there is nothing to be done about public acceptance of this erosion of the rule of law. Over two centuries of Supreme Court rulings provide convincing evidence that the Justices have political biases that influence how they rule, and voters have long understood that presidential elections matter to the future of the courts. But these realities do not mean we should be indifferent to judicial actions that undercut the rule of law. Rather it means that the courts need to do better at suppressing their partisanship.

Perhaps we should celebrate that partisanship is obvious in only a small fraction of Supreme Court rulings, but there is a commonality among those cases in which politics play a central role. As noted above, most of the cases in which the Justices reliably divide along political lines involve rights claims—usually expressly in the context of a constitutional rights guarantee, and sometimes indirectly in the contexts of the separation of powers and federalism. Here, the Justices on both sides are divided by constricted understandings of individual liberty.

Liberal justices are quick to defend liberties of conscience, personal autonomy, and criminal due process while granting the state expansive powers over economic liberties. Conservative justices reliably come to the defense of economic liberties and the right to bear arms while allowing for extensive government interference with private thought and action. There are exceptions, of course, and the differentiation of the two positions varies over time. But the important point is that whatever the differentiation is, it has no basis in the Constitution. There is nothing in the Constitution allowing that some rights warrant greater protection than others. While the disagreement over the legitimacy of unenumerated rights has tended to reflect party lines, it is a dispute over the existence of particular rights not the relative importance of acknowledged rights. If the Justices on both sides would recognize that the Constitution guarantees all liberties equally, there would be neither liberal nor conservative Justices, but only Justices who stand in defense of individual liberty and limited government.

The coming presidential election will have a significant impact on the future of our constitutional freedoms. There is little doubt that the electorate understands and has come to accept that truth. Only the Justices of the Supreme Court and other judges can counter this unfortunate reality of a politicized judiciary. Political bias will creep inevitably into some judicial decisions. Judges are not automatons. But the rule of law and constitutional government depend on an independent judiciary, which in turn depends on judges committed to nonpartisanship and a public that demands such commitment.