The Obama administration signalled for the first time on Monday that it was willing to enter into discussions with Iran over the advance of Islamist insurgents in Iraq, but stressed that any contact between the two adversaries would be informal and rejected an earlier suggestion by secretary of state John Kerry that military collaboration was a possibility.

In an indication of how sensitive in Washington any such cooperation would be, officials quickly rowed back from remarks by Kerry, who had declined in an interview to rule out military cooperation with Tehran. But officials later insisted that any contact would be limited to informal discussions that would take place on the margins of nuclear talks in Vienna.

Fighters from the Islamic State in Iraq and the Levant (Isis) have rapidly advanced through mostly Sunni areas of Iraq in recent days, capturing several cities. It was reported on Monday that they had taken Tal Afar, a northern Iraqi city. On Sunday, the insurgent fighters posted images purporting to show the execution of hundreds of Shia fighters.

More than a decade after the US invasion, there is concern in Washington that Iraq is once against unravelling, and a recognition in some quarters that Iran's involvement in shoring up the government in Baghdad is inevitable. But Kerry went further than his administration colleagues in entertaining military cooperation with Iran against a common adversary.

"We're open to discussions if there is something constructive that can be contributed by Iran, if Iran is prepared to do something that is going to respect the integrity and sovereignty of Iraq and ability of the government to reform," Kerry told Yahoo News.

Pressed by interviewer Katie Couric over whether that would include military cooperation, Kerry replied: "At this moment I think we need to go step-by-step and see what in fact might be a reality. But I wouldn't rule out anything that would be constructive to providing real stability."

Less than three hours later, the Pentagon released a series of public statements that firmly ruled out military coordination. "There has been no contact, nor are there plans for contact, between [the Department of Defense] and the Iranian military on the security situation in Iraq," commander Bill Speaks, a Pentagon spokesman, told the Guardian.

Officials said any cooperation would be limited to discussions on the sidelines of talks Vienna between US and Iranian diplomats, and would not include military coordination. "It's possible that on the sidelines of those [nuclear] discussions there could be discussions surrounding the situation in Iraq," Pentagon spokesman Rear Admiral John Kirby told reporters, according to Reuters.

"But there is absolutely no intention and no plan to coordinate military activity between the United States and Iran … there are no plans to have consultations with Iran about military activities in Iraq."

The State Department also took the unusual move of walking back the remarks of Kerry. Spokeswoman Jen Psaki said that while there may be discussions about the political situation in Iraq, "We're not talking about coordinating any military action with Iran."

But the prospect of diplomatic, let alone military, cooperation between Washington and Tehran over the chaos in Iraq represents a dramatic turnaround for the two rival powers, whose relations, frozen for several decades, have only begun to thaw over the past year.

Military experts say any US air strikes in Iraq would will be impeded by the lack of intelligence from the the ground. An Iranian offensive, by contrast, would be expected to involve elite forces of ground troops that would engage in direct combat with Isis fighters, gaining a detailed knowledge of the battle lines.

Iran's president, Hassan Rouhani, has not ruled out working with the US against Isis in Iraq. "We can think about it if we see America confronting the terrorist groups in Iraq or elsewhere," he told a press conference on Saturday.

Yet the notion of a partnership between the longtime foes prompted intense resistance in some quarters of Washington and Tehran on Monday. "It would be the height of folly to believe that the Iranian regime can be our partner in managing the deteriorating security situation in Iraq," senator John McCain said in a statement.

McCain's remarks contrasted with those of another Republican hawk, Lindsey Graham, who on Sunday expressed support for cooperating with Iran. McCain and Graham are usually in lockstep over foreign policy issues and their dispute revealed the divisions uncovered by the prospect of a collaboration with Iran.

In Tehran, a number of senior Iranian officials also played down the prospects of cooperation. "The Islamic Republic of Iran has had no negotiations with the Americans over mutual cooperation in Iraq," said deputy foreign minister Hossein Amir-Abdollahian.

Washington has dispatched some of its most senior White House and State Department officials to the nuclear talks in Austria, including the top deputy secretary of state, William Burns. He was scheduled to meet Iran's foreign minister Mohammad Javad Zarif and EU foreign policy chief Catherine Ashton on Monday.

Their meeting kicks off five days of negotiations between Iran and the six world powers collectively referred to as "P5+1". Before arriving in Vienna, Zarif spoke by telephone with the British foreign secretary, William Hague, about the possible role Iran could play in easing the conflict in Iraq.

Any substantial conversation about cooperation in Iraq would be highly unusual, and contrast with the US refusal to allow Iran to be part of peace talks over the civil war in Syria. It could have major regional repercussions and would be likely to infuriate Saudi Arabia.

A senior US administration official, speaking to reporters on the condition of anonymity, said Washington and Tehran had previously held discussions over Afghanistan and communicated on other issues at the United Nations.

"There have been times when it made sense to be part of a conversation," the official said, before playing down the suggestion that cooperation over Iraq would significantly alter relations between the two countries.

"No one should expect that all of a sudden, overnight, even if we resolve the nuclear agreement, that everything will change. It will not," the official said. "The fundamentals remain exactly as they are. Until we resolve the nuclear issue there cannot be any kind of fundamental change in this relationship."

In Iraq on Monday, the capital, Baghdad, remained outside the grasp of Isis. But the mayor of Tal Afar, a city of 200,000 people located 260 miles north-west of Baghdad, told the Associated Press that the insurgent group was in control there. A resident said militants in pickup trucks with machine guns and jihadi banners were roaming the streets as gunfire rang out.

Fighting in Tal Afar began on Sunday, with Iraqi government officials saying Isis fighters were firing rockets seized from military arms depots in the Mosul area. They said the local garrison suffered heavy casualties and the main hospital was unable to cope with the wounded.

There were fears that militants would carry out further atrocities in Tal Afar, which is ethnically mixed and made up of Shias and Sunni Turkomen.

Claims at the weekend that the insurgents had killed 1,700 Iraqi soldiers could not be verified. But pictures, on a militant website, appear to show masked Isis fighters loading captives on to flatbed trucks before forcing them to lie facedown in a shallow ditch with their arms tied behind their backs. The final images show the bodies of the captives soaked in blood after being shot at several locations.

Iraq's chief military spokesman, Lt Gen Qassim al-Moussawi, said the photos were genuine and that he was aware of cases of mass murder of captured Iraqi soldiers in areas held by Isis.

Tal Afar's capture came hours after Nouri al-Maliki, the Iraqi prime minister, vowed to retake every inch of territory seized by the militants.

"We will march and liberate every inch they defaced, from the country's northernmost point to the southernmost point," Maliki told volunteers joining up to fight the insurgents.

Additional reporting by Mark Tran in London

• This story was amended on 17 June to correct the rank of commander Bill Speaks, a Pentagon spokesman.

