MANILA—Philippine President Rodrigo Duterte is facing the biggest test to his authority since taking office as his attempt to force the arrest of a leading critic has run into unexpected resistance from institutions that had previously done his bidding.

The opponent, Sen. Antonio Trillanes, has been holed up at his office in the Senate for nearly a week for fear of being arrested if he steps outside. A series of judicial rulings in coming days could determine whether there is a quick arrest, triggering potential street unrest,...