Iraq's embattled prime minister, Nouri al-Maliki, appeared to have lost his job on Monday, after the country's president appointed a rival Shia candidate to form a new government in a bid to end the deadlock that has paralysed the Baghdad government while jihadists have swept through the country's north.

Maliki had seemed to be clinging to his post, but he was abandoned by party allies and sidelined by religious and regional backers who no longer believe he can save the crumbling state.

His defiance sets the scene for yet another volatile period in Iraqi politics at a time when the Islamic State (Isis) jihadist group continues to rampage through the country, fast diminishing the authority of the central government. It also adds more uncertainty to a pivotal period in the modern history of the region, with the unitary borders of Iraq and its neighbours under mounting pressure to hold together.

Iraq's military leadership was being closely watched by regional players on Monday. The US warned military officials not to get involved in the political process.

The US government said that it would arm Iraqi Kurdish militias to prevent the fall of the final bastion of pro-US territory in Iraq, while Britain is deploying RAF Tornado jets to provide greater surveillance in the north of the country.

The move to effectively remove Maliki came after the White House launched a fresh volley of phone calls to politicians in Baghdad on Monday. In his fourth diplomatic intervention in as many days, the US vice-president, Joe Biden, called both the Iraqi president, Fouad Massoum, and the man he has selected to try to replace Maliki, Haider al-Abadi, indicating that further US military support could follow swiftly if they succeed in their efforts to form a new government.

On Monday night US president Barack Obama welcomed new leadership in Iraq as "a promising step forward", saying the only lasting solution is the formation of an inclusive government. Obama did not mention Maliki but called for Iraqi political leaders to work peacefully through a political transition. "These have been difficult days in Iraq," Obama said outside his rented vacation home on Martha's Vineyard. "I'm sure there are going to be difficult days ahead."

In what amounted to a revolt within his own Dawa party, Massoum announced that Abadi, a long-time ally of Maliki, would be nominated as prime minister. The nomination makes it all but impossible for Maliki, who was backed by Washington during his last two terms as leader, to assemble a government. However, his supporters angrily vowed to oppose Abadi's appointment, claiming it had no legitimacy and insisting Maliki should be called to form a government because his bloc won most seats in the 30 April election.

Maliki's dramatic fall from grace came hours after he gave a midnight press conference on Sunday accusing Massoum of violating the constitution by failing to name a prime minister within 15 days. Pointedly, he also sent Iraq government forces on Sunday evening to the green zone, the home of Iraq's government, and to the president's residence.

The international community has repeatedly put pressure on Maliki to step down ever since the start of the jihadist insurrection in June that saw Mosul, Tikrit and much of western Iraq fall to Isis and Kirkuk fall to the Kurds.

Iraq's army abandoned its posts in the north in one of the more spectacular routs in modern military history. And, as commander in chief, Maliki had been unable to reverse the losses.

The US and Britain, along with Saudi Arabia and the Gulf states, claim that overtly sectarian policies adopted by Maliki since the onset of the civil war in Syria have disenfranchised the country's Sunnis, in part making them prey for the jihadists who now control much of Sunni Iraq.

The powerful Shia cleric, Moqtada al-Sadr, whose movement controls dozens of seats in parliament, expressed his support for Abadi's nomination, describing it as the "first sign" the country was headed back to safety. "I think that this nomination will be an important start in order to end the crisis that the people are undergoing such as security and service problems," he said in a statement.

Under Iraq's constitution, Maliki remains prime minister for the next 30 days, until a new cabinet is formed. His son-in-law, Hussein al-Maliki, said Maliki would seek to overturn Abadi's nomination in the courts. "We will not stay silent. The nomination is illegal and a breach of the constitution. We will go to the federal court to object," he told Reuters.

Haider al-Abadi graduated from Manchester University in 1981. Photograph: Ahmed Saad/Reuters

Abadi, the first deputy speaker of Iraq's parliament, is a member of Maliki's party, which in recent weeks has turned against the prime minister.

State TV showed footage of President Massoum shaking hands with Abadi and telling him: "I hope you will be successful in forming a broader-based government."

Abadi is a low-key figure who lived in Britain and graduated in 1981 with a doctorate from Manchester University. According to his Facebook biography, his favourite quotation is "the key to leadership is tolerance". An electrical engineer and businessman, Abadi entered politics after the 2003 US-led invasion of Iraq. He served as the head of the parliament's finance committee, a political adviser to the prime minister and a minister of communications.

The new prime minister swiftly called on Iraqis to unite against the "barbaric" campaign waged by Isis. "We all have to co-operate to stand against this terrorist campaign launched on Iraq and to stop all terrorist groups," Abadi said in remarks broadcast on state television just after the president had asked him to form a government, according to Reuters.

Pointedly referring to al-Abadi as the Iraqi prime minister-designate, a White House spokesman said: "The vice-president relayed President Obama's congratulations and restated his commitment to fully support a new and inclusive Iraqi government, particularly in its fight against Isil."

Earlier the US secretary of state, John Kerry, said the US stands "absolutely squarely behind President Massoum", but urged restraint amid reports that Maliki may resist what he called a "coup" against his government. "What we urge the people of Iraq to do is to be calm," Kerry said. "There should be no use of force, no introduction of troops or militias into this moment of democracy for Iraq."

Speaking in Australia earlier on Monday,Kerry had warned Maliki to abide by the constitutional process and not to use his powers as head of the armed forces to cling to office.

Kerry said that any move to circumvent the political process would lead to a cut-off of international aid. He said: "There should be no use of force, no introduction of troops or militias into this moment of democracy for Iraq."

On Monday, former US officials in Washington also cautioned that the appointment of Abadi, though supported by the White House, was not yet a definitive sign of the more inclusive Iraqi government that Obama has been calling for.

"The prime minister still has to be able to form a government, so I am not sure it is really game over," said Jon Alterman, Middle East programme director at the Center for Strategic and International Studies. "Nouri al-Maliki is a hard knuckled veteran of all of this and he is not afraid to roll the dice … he is willing to take risks."

"Maliki knows it is very difficult to gain a third term and is playing a high-stakes game to try and ensure his authority and influence continue into the new government, despite who may officially become prime minister," Kamran Bokhari, a Middle East specialist at analysis firm Stratfor, told the Associated Press.

Other analysts said that Maliki is increasingly likely to use force to protect his political position. They add that he has consolidated control over the security apparatus by establishing extra-constitutional security bodies. There was now a direct chain of command from commanders to his office, they said. Meda Al Rowas, an analyst at IHS Country Risk said Maliki's rivals might seek to remove him using force, raising the prospect of Shia infighting within Baghdad, as well as in southern provinces such as Basra, Najaf and Karbala.