The Australian foreign minister said on Sunday she was quitting after a failed bid for the nation's top job during a messy party-room coup, as the new prime minister announced a peace-making Cabinet.

The deputy chief of the Liberal Party, Julie Bishop had put her hand up to be one of three candidates to replace former prime minister Malcolm Turnbull in Friday's leadership challenge, but received minimal support from colleagues even as opinion polls pointed to her popularity among voters.

The departure of Ms Bishop, a rare female voice in the Australian government, has raised questions about whether she fell victim to party politics and a perceived glass ceiling for women in Canberra.

Ms Bishop said: "I will remain on the backbench as a strong voice for Western Australia."

A moderate, she reportedly garnered only 11 votes out of 85 in the leadership ballot - significantly lower than the two other rightwing challengers, coup instigator Home Affairs Minister Peter Dutton and Treasurer Scott Morrison.

A leaked WhatsApp chat between some Liberal members, revealed by the Australian Broadcasting Corporation Sunday, showed them pushing against voting for Ms Bishop as a tactic to back Mr Morrison, who finally emerged as the winner.

Australia has endured a turbulent period in politics that has seen six changes in the top job in 11 years.