JAKARTA, Indonesia—A bid by religious activists to outlaw sex outside marriage and gay sex has been narrowly rejected by a powerful court—a rare recent victory for Indonesia’s moderate forces.

The court’s decision bolstered social liberals in the world’s most populous Muslim-majority nation. Indonesia has taken a conservative bent in recent years, with the government cracking down on alcohol sales and banning some pornographic online content, as conservative Islam plays an increasingly important role in politics.

In...