No matter who Donald Trump picks to replace Anthony Kennedy on the Supreme Court, the nation’s high court is going to move to the right on a lot of issues — chief among them abortion rights, which Kennedy largely affirmed during his time on the Court. How far right will be up to Chief Justice John Roberts, who many are calling the Court’s new swing vote.

The data bears it out: FiveThirtyEight calculated that, safely assuming Trump picks a new justice who is more conservative than George W. Bush nominee Roberts (and all the shortlisted justices are), then the chief justice will be the new median judge on the Court — becoming the fifth vote in those narrow 5-4 decisions.

Roberts will likely be the key vote in deciding how far the Supreme Court is willing to roll back Roe v. Wade, the 1973 Supreme Court decision that guaranteed women a basic right of access to abortion, once Kennedy’s replacement is seated and the right case comes before the justices.

Though all justices try to give a politically palatable version of their answer, Roberts was remarkably difficult to pin down on abortion rights during his Senate confirmation hearings. “There’s nothing in my personal views based on faith or other sources that would prevent me from applying the precedents of the Court faithfully,” he told the senators back in 2005.

To understand what he thinks, we have to look back further: As a federal attorney in Republican administrations, Roberts helped craft briefs that said Roe had been wrongly decided and should be overturned. While the chief justice is widely said to be deferential to existing precedents — that was the crux of his statements on Roe before the Senate — he has also in his opinions helped to roll back existing case law.

Lastly, Roberts is said to be mindful of the Court’s institutional reputation. That may be one reason for the chief justice to avoid overturning Roe all at once. But he should have plenty of opportunities to chip away at abortion rights until there is little left, a playbook that he’s very recently deployed on other high-stakes issues.

A Supreme Court with John Roberts as the swing vote on abortion is one that should be ready and willing to uphold ever-increasing restrictions on abortion rights. The only real question is whether they’d be so overt as to say outright that they are overturning Roe in the process.

What John Roberts has actually said about abortion

Roberts has been very guarded in the public record in his statements about abortion. During his Senate hearings, his stock answer was that Roe v. Wade was settled law and that he had a healthy respect for establishment precedents.

“Precedent plays an important role in promoting stability and evenhandedness,” he said then. “It is not enough that you may think the prior decision was wrongly decided.”

To divine more about the chief justice’s views on abortion requires some interpretation.

“We know very little about Chief Justice Roberts’ personal views on abortion and reproductive freedom. What we do know is that he is a practicing Catholic and his wife (who is also a Catholic) has been involved with Feminists for Life, a group that opposes abortion,” Melissa Murray, a law professor at New York University, told me by email. “And in 1990, as a lawyer with the Department of Justice, Roberts penned a brief that argued Roe v. Wade was wrongly decided and should be overturned.”

That brief — which Roberts helped write as deputy solicitor general in 1990, according to the National Women’s Law Center (NWLC) — is Exhibit A for abortion rights supporters who fear that Roberts would overturn Roe. The brief argued that Roe v. Wade “was wrongly decided and should be overruled” and that a right to an abortion “find[s] no support in the text, structure or history of the Constitution.”

While one cannot automatically assume that a political appointee agrees with a brief they are asked to write, the NWLC said there was circumstantial evidence that suggests the anti-Roe position was consistent with Roberts’s own.

The Los Angeles Times reported in 2005 that Roberts had led a “small team of conservative lawyers who were determined to overturn Roe vs. Wade” at the Justice Department. Scott Ruskay-Kidd, senior staff attorney with the Center for Reproductive Rights, also noted that the position Roberts held in the Justice Department is “generally assumed” to be a political appointee with some policymaking discretion and that, during Roberts’s tenure, the federal government “consistently argued that the Supreme Court should overrule Roe v. Wade.”

But it’s one thing to think the Court should overturn Roe — it’s another to be the person actually do it.

Would precedent or reputation really stop Roberts from overturning Roe?

The swing vote on the Supreme Court is moving from Kennedy — who dramatically voted to uphold the main precepts of Roe in 1992’s Planned Parenthood v. Casey — over to Roberts, who at least appears to have some hostility toward abortion rights.

The other two variables are Roberts’s alleged deference to precedents like Roe and his cognizance of the court’s reputational standing, particularly when two-thirds of Americans say they do not want to see Roe overturned.

“As chief justice, Roberts has evinced particular concerns about overruling settled precedents,” Murray said, and “as chief justice, he has been especially concerned about preserving the Court’s legitimacy and public stature.”

There are undoubtedly times when the chief justice lives up to those standards. He crossed ideological lines and upheld the Affordable Care Act in 2012 and voted in the law’s favor in a second case a few years later.

But there are also unavoidable examples when he did not strictly follow those instincts. On collective bargaining rights and on civil rights most memorably, Roberts has voted to invalidate prior Court decisions, striking conservative victories against labor unions and for voting restrictions.

“This is all to say that while Roberts might be wary of voting to overturn Roe outright ... he might be more amenable to continuing to chip away at the abortion right, allowing the Court, over time, to eventually overturn Roe entirely,” Murray said.

On abortion specifically, he has voted in favor of restrictions since taking a seat on the Supreme Court. In 2007, Roberts voted to uphold a late-term abortion ban. He was in the minority on Whole Woman’s Health v. Hellersedt, when Kennedy sided with the Court’s liberals to rule that states could not place an undue burden on a woman’s access to abortion. He also voted last month along with the Court’s other four Republican appointees to overturn a recent California law that regulated anti-abortion crisis pregnancy centers.

“That is the most probative information of how he ruled in abortion cases and may rule in the future,” Ruskay-Kidd said.

The most Robertsian path: gradually restricting abortion more and more

Combine all those ingredients and the most likely path under Roberts seems to be an incremental process of narrowing abortion rights piece by piece. Roberts’s reputational concerns are real, and the public seems more in favor of keeping Roe than nixing it. The chief justice also does have legitimate philosophical priors about respecting precedent and issuing the narrowest possible opinions. A splashy, dramatic, and wholesale reversal of Roe is not really in keeping with Roberts’s modus operandi.

But a more gradual rollback is certainly plausible, as Irin Carmon has outlined in the Washington Post. In fact, we’ve seen Roberts deploy that strategy twice in two other highly contested areas of law, letting a series of smaller opinions lead him toward overturning the underlying precedent that had been in question.

On labor rights, Roberts used several narrower rulings to curtail the collective bargaining rights guaranteed in part by 1977’s Abood v. Detroit Board of Education before he and a Court majority reversed Abood entirely this year. With the Voting Rights Act, as Carmon observed, Roberts started with a pretty narrow decision in 2009 before ruling to overturn the heart of the law a few years later in Shelby County v. Holder.

“If a newly constituted Roberts Court doesn’t openly overrule Roe, it could accomplish much the same through a stealth overruling — by allowing hostile and restrictive regulation of abortion without announcing that the Court was overruling prior precedent,” Reva Siegel, a Yale University law professor who wrote on this issue for the New York Times, told me.

Republican-led states are putting forth plenty of policies — six-week bans, administrative requirements for abortion providers, and other restrictions to states’ abortion access — that could be used to restrict abortion access and stand under a Robertsian legal theory. In states like Kentucky and Mississippi, which have already added abortion restrictions to the point that there are only one or two health care providers in the state who perform them, women could very quickly for all intents and purposes be living in a post-Roe world.

The threat to Roe is very real. It will be up to John Roberts to decide what precise form it takes.