SYDNEY—Australia’s Prime Minister Malcolm Turnbull survived a leadership challenge on Tuesday, as he struggles to pass key pieces of his legislative agenda and keep members of his party in line.

The challenge from Peter Dutton, which Mr. Turnbull survived by a vote of 48 to 35, is a reminder of the nagging political instability in Australia, which has changed prime ministers five times in the past 10 years. No leader in the past decade has survived a three-year term without being ousted by their own party.

A...