Around 2 a.m. Thursday, as it became evident that Malaysia’s effective one-party state had come to an end after 61 years, Prime Minister Najib Razak was in shock.

At his private residence in Kuala Lumpur, surrounded by family and close aides, Mr. Najib chewed over the events of the past 24 hours, when Malaysians voted in droves for the opposition—bucking poll predictions and knocking the ruling party out of power it had held since independence from the U.K. in 1957.

As...