Now that he’s officially taken his seat on the Supreme Court, Brett Kavanaugh has no obvious reason to care what you think. Neither does Sonia Sotomayor, or Samuel Alito, or Ruth Bader Ginsburg. They and their colleagues are justices for life, which should in theory give them the freedom to write unpopular opinions.

But Supreme Court history shows that’s not always how it works. In the past, the justices have appeared to bend to popular opinion, in addition to being reined in by other branches of government when they deviate dramatically from the mainstream. That history has a lot to tell us about how much leeway the court’s new majority has when deciding future cases on issues where a conservative ruling might spark a backlash, like abortion. These justices may have an unprecedented opportunity to shift an already conservative court even further to the right, but they’ll likely have to navigate more than just jurisprudence if they want their rulings to last.

The relationship between the court and the rest of us is well-studied by historians and political scientists. And several studies do suggest that the justices respond to public opinion. For example, Peter Enns, a political science professor at Cornell University, found that the court’s ideological tilt tracks with public opinion over time. “We can’t get inside their minds and understand how they’re weighing the potential public reaction,” he said. “But when the public’s perspective is more liberal, we consistently see more liberal Supreme Court decisions, and the reverse is true when the public mood is more conservative. It’s hard to believe that’s just a coincidence.”

It’s possible, of course, that the justices’ individual worldviews are simply influenced by the same forces that shape broader public opinion. But history has shown that there are practical reasons for the court to avoid bucking mainstream sentiment. In the past, Congress, the president and state governments have openly defied controversial Supreme Court rulings. Congress can also regulate the types of cases the court is allowed to hear or dilute a recalcitrant majority by “packing” the court with ideologically sympathetic justices. Proposals that take advantage of that power have been considered seriously only a handful of times, according to Tom Clark, who is a political scientist at Emory University and studies the limits of judicial independence. But when they have, the court avoided formal retaliation — like being remade into a 15-member chamber — because the justices ultimately backed down.

Perhaps the most famous example of a Supreme Court brought to task by the other branches of government was in the 1930s, which also happened to be the last time the court was controlled by a strong conservative majority. The country was in the depths of the Great Depression, and the Supreme Court was aggressively striking down President Franklin D. Roosevelt’s progressive economic legislation, which was widely popular at the time. Finally, Roosevelt announced a plan to increase the size of the court by as many as six justices. The scheme ultimately collapsed in Congress — and may have done some damage to Roosevelt’s popularity in the process — but not before one of the right-leaning justices suddenly began voting to uphold New Deal laws that were identical to ones he had voted to gut only a year earlier.

Barry Friedman, who is a professor at New York University Law School and studies legal history, said there’s a clear lesson from the 1937 court-packing episode. “The court can’t get too far out of step with public opinion before something happens to rein them in,” he said.

And it wasn’t the first time Congress or the president had intervened when the court appeared to block a popular policy agenda. During Republican-led Reconstruction, when three constitutional amendments were passed to end slavery, give legal equality to former slaves and prohibit racial discrimination at the polls, the GOP swept the 1866 congressional elections. That gave them a veto-proof majority against President Abraham Lincoln’s Democratic successor, who wanted to allow Southern states to re-enter the union more easily than many Republicans were willing to countenance. When the Supreme Court seemed likely to halt Reconstruction’s progress, Congress repeatedly changed the size of the court for political ends and revoked the court’s ability to review a case that could have threatened military rule in the South — a decision the court itself upheld.

Even in moments when the court has taken steps to shore up a controversial decision, a backlash has first delayed the enforcement of the ruling and eventually set the stage for the court to back down. The landmark ruling in Brown v. Board of Education, which said that school segregation laws were unconstitutional, was decided unanimously with the explicit goal of lending additional legitimacy to the decision. But it was still met by outrage and defiance from Southern state governors and a lukewarm response from President Dwight Eisenhower, who thought the court should pursue integration by subtler means. And after years of striking down attempts to thwart desegregation — and amid a fierce national debate about the use of school busing to integrate schools — the court finally capitulated to the status quo when it ruled in 1974 that the Detroit public schools could remain functionally segregated. In a dissenting opinion, Justice Thurgood Marshall wrote that the majority’s ruling was “more a reflection of a perceived public mood” that desegregation had gone far enough than “the product of neutral principle of law.”

These historical showdowns are uncommon, but Clark said that’s precisely because the Supreme Court justices are concerned about their own institutional legitimacy and aware of limitations on their power. For a book published in 2010, Clark reviewed all of the bills introduced in Congress to curb the court starting in 1877 and found that the court seemed to respond through its opinions. When Congress introduced more bills that would limit the court’s power, the court struck down fewer laws. Clark sees the court-curbing bills — which almost never became law — as a way of sending a signal to the court. “These reactions allow the justices to learn if they’ve gone too far out of line,” he said.

Whether the current Supreme Court justices have internalized this view is another question. Some research has concluded that the court’s historical “swing” justices are more likely to respond to strategic concerns like public opinion when casting pivotal votes. If true, the addition of Kavanaugh could put even more pressure on the new median justice, Chief Justice John Roberts, who is already known for his concern about the court’s reputation, to moderate the court’s right wing. He might even join the liberals in key cases, as retired Justice Anthony Kennedy, the court’s previous median justice, sometimes did.

Who controls Congress could have an impact on the conservative majority’s willingness to make a sharp right turn, too. If Republicans retain control after this year’s midterm elections, the court would be largely safe from reprisal (although its institutional legitimacy could be even further damaged on the left). But the stakes would change considerably if the Democrats take one or both houses of Congress in November.

Part of the problem is that no one — including the justices — knows exactly what “too conservative” means. Over the past decade, the Roberts court has already issued a slew of right-leaning rulings without triggering widespread public outrage. And even when a decision is unpopular, it can be difficult to predict what will spur Congress and the president to action and what won’t. The 2010 ruling in Citizens United v. Federal Election Commission, which allowed unlimited corporate spending on direct advocacy for and against political candidates, was broadly disliked when it came down, but the Democrats — who were in control of both Congress and the White House at the time — didn’t retaliate against the court.

The country’s deep ideological divisions may also help insulate the court, said Steve Vladeck, a law professor at the University of Texas. “We’re so divided — it’s hard to think of many issues that an outright majority would get really angry about,” he said. Overturning Roe v. Wade is one clear example of a ruling that could spark a legitimacy crisis, since polls have consistently found that a solid majority of Americans oppose such a move. But it’s possible to significantly undermine abortion rights without overruling Roe explicitly, using what some legal experts have called the “death by a thousand cuts” approach.

These questions won’t be answered overnight. So far, the Supreme Court’s term looks relatively sleepy, and it will take time for the engines of the conservative legal movement, now emboldened by Kavanaugh’s confirmation, to bring new, sweeping challenges to the court. But the figure to watch for clues isn’t Kavanaugh — it’s Roberts, who will need to start figuring out what kind of conservative court he wants to lead.