Obama presented the troop withdrawal as the fulfilment of one of his election promises Reuters

The US suffered a major diplomatic and military rebuff on Friday when Iraq finally rejected its pleas to maintain bases in the country beyond this year.

Barack Obama announced at a White House press conference that all American troops will leave Iraq by the end of December, a decision forced by the final collapse of lengthy talks between the US and the Iraqi government on the issue.

The Iraqi decision is a boost to Iran, which has close ties with many members of the Iraqi government and which had been battling against the establishment of permanent American bases.

Obama attempted to make the most of it by presenting the withdrawal as the fulfilment of one of his election promises.

"Today I can report that, as promised, the rest of our troops in Iraq will come home by the end of the year. After nearly nine years, America's war in Iraq will be over," he told reporters.

But he had already announced this earlier this year, and the real significance today was in the failure of Obama, in spite of the cost to the US in dollars and deaths, to persuade the Iraqi prime minister Nouri al-Maliki to allow one or more American bases to be kept in the country.

Obama was formally told of Maliki's final decision on Friday morning in a video conference.

Speaking later to reporters, Obama glossed over the rejection, describing it as Iraq shaping its own future.

He told reporters that the "tide of war is receding", not only in Iraq but in Afghanistan and in Libya.

"The United States is moving forward to a position of strength. The long war in Iraq will come to an end by the end of this year. The transition in Afghanistan is moving forward and our troops are finally coming home," he said.

Obama rose to political prominence on the back of his opposition to the Iraq war.

"Over the next two months, our troops in Iraq, tens of thousands of them, will pack up their gear and board convoys for the journey home," he said.

"The last American soldier will cross the border out of Iraq with their heads held high, proud of their success, and knowing that the American people stand united in our support for our troops," he said. "That is how America's military efforts in Iraq will end."

But Republicans criticised the failure to secure a deal with the Iraqis, describing it as a setback for the US.

John McCain, one of the leading foreign affairs specialists in the Senate and Obama's Republican opponent in the 2008 White House race, said: "Today marks a harmful and sad setback for the United States in the world. I respectfully disagree with the president: this decision will be viewed as a strategic victory for our enemies in the Middle East, especially the Iranian regime, which has worked relentlessly to ensure a full withdrawal of U.S. troops from Iraq."

Mitt Romney, front-runner in the race to take on Obama in the 2012 White House race, said: "The unavoidable question is whether this decision is the result of a naked political calculation or simply sheer ineptitude in negotiations with the Iraqi government."

One of the sticking points in the negotiations with Iraq was a US demand that American forces remaining in the country after December would enjoy the same immunity from prosecution as they do now. The Iraqi government, conscious of public anger over many controversial incidents involving US troops and defence contractors over the last decade, refused.

The Pentagon had wanted the bases to help counter growing Iranian influence in the Middle East. Just a few years ago, the US had plans for leaving behind four large bases but, in the face of Iraqi resistance, this plan had to be scaled down this year to a force of 10,000. But even this proved too much for the Iraqis.

Denis McDonough, the White House deputy national security adviser, speaking to reporters after Obama's press conference, denied that the withdrawal was a sign of growing Iranian influence.

"You see an Iran that is weaker and more isolated," he said, noting various incidents such as a sense of international outrage over an alleged plot by Iran to assassinate the Saudi ambassador to Washington.

Although the US is pulling out all troops, it will keep its embassy in Baghdad and two consulates. There will also be about 4,000-5,000 defence contractors, White House aides said.

Since the invasion in 2003, 1 million members of the US military have been deployed to Iraq, of whom 4,482 have been killed and 32,200 wounded.

Obama said there were 180,000 troops deployed in Iraq and Afghanistan when he took office in January 2009, and that number has been halved and will continue to fall.

A few US military personnel will be based in Iraq temporarily from time to time, just as they are in other countries with links to the US such as Egypt and Jordan, White House aides said. These would primarily be trainers helping out with new equipment bought from the US, such as F-16 fighters Iraq purchased last month.

Maliki, though he has been criticised in the past for being too close to Iran, had wanted to keep some US troops in Iraq to help train Iraqi security forces and to help in the event of a resurgence of sectarian violence. But he had to bow to pressure from pro-Iranian politicians and others in his coaliton government who wanted all US troops out.

Obama was ambivalent on the issue, seeing a total withdrawal as a good sell to a US public tired of war. But the Pentagon had wanted the bases, and the president reluctantly sided with the military staff.

It will be a major logistical exercise, moving not only the remaining 39,000 US troops but mountains of equipment from bases that are the size of small American suburbs, complete with coffee-shops, bowling alleys and cinemas.

The Pentagon is wary of a final attack as the final pullout gets under way.

• This article was corrected on 23 October 2011 because it described Nouri al-Maliki as Iraqi's president instead of as its prime minister, and at one point said he had wanted to keep some US troops "in Iran".