WASHINGTON — In his first 13 years on the Supreme Court, Chief Justice John G. Roberts Jr.’s main challenge was trying to assemble five votes to move the court to the right, though there were only four reliably conservative justices.

Now he faces a very different problem. With the retirement of Justice Anthony M. Kennedy and his replacement by Justice Brett M. Kavanaugh, the chief justice has the votes he needs on issues like abortion, racial discrimination, religion and voting. At the same time, he has taken Justice Kennedy’s place as the swing vote at the court’s ideological center, making him the most powerful chief justice in 80 years.

But all of that new power comes at a dangerous time for the court, whose legitimacy depends on the public perception that it is not a partisan institution. “We don’t work as Democrats or Republicans,” Chief Justice Roberts said in 2016, and he reiterated that position in an extraordinary rebuke of President Trump last month.

He seemed to underscore that point again on Friday, joining the court’s four-member liberal wing, all appointed by Democratic presidents, to reject a request from the Trump administration in a case that could upend decades of asylum policy. This month, he drew sharp criticism from three conservative colleagues for voting to deny review in two cases on efforts to stop payments to Planned Parenthood.