The United States has expressed horror at an enormous cache of photographs documenting how the Syrian regime has killed an estimated 11,000 detainees, warning that the evidence of war crimes has cast a shadow over imminent peace talks.

An inquiry by three former prosecutors who reviewed some 55,000 images smuggled out of the country revealed the “systematic torture and killing” of prisoners held by Bashar al-Assad’s forces. Their report was made public by the Guardian and CNN on Monday.

“We stand with the rest of the world in horror at these images which have come to light,” a US administration official told the Guardian. “We condemn in the strongest possible terms the actions of the regime and call on it to adhere to international obligations with respect to the treatment of prisoners.

“We have long spoken out about mistreatment and deteriorating prison conditions in Syria,” the official added. “These latest reports, and the photographs that support them, demonstrate just how far the regime is willing to go to not only deny freedom and dignity to the Syrian people, but to inflict significant emotional and physical pain in the process. To be sure, these reports suggest widespread and apparently systematic violations of international humanitarian law.”

William Hague, the British foreign secretary, said the report "offers further evidence of the systematic violence and brutality being visited upon the people of Syria by the Assad regime". He said Britain would continue to press for "action on all human rights violations in Syria" and for accountability for those who perpetrate them.

The three former prosecutors, who had experience of criminal tribunals for the former Yugoslavia and Sierra Leone, examined thousands of Syrian government photographs and files recording deaths in the custody of regime security forces between March 2011 and August 2013.

Most of the victims were young men and many corpses were emaciated, bloodstained and bore signs of torture. Some had no eyes; others showed signs of strangulation or electrocution.

The publication of the report on Monday came ahead of talks between the opposing sides in Syria, which will begin on Wednesday in the Swiss town of Montreux.

The US official said the latest evidence of atrocities "tarnished the environment" for those negotiations, which will see opposition forces engage in direct talks with the regime for the first time since the 34-month conflict began.

“This latest report of horrific and inhumane prison conditions/actions further underscores that, if anything, it is tarnishing the environment for the talks,” the official said.

Those talks had already been thrown into chaos earlier on Monday when Ban Ki-moon, the UN secretary general, was forced to rescind an invitation to Iran to participate in the negotiations.

Tehran had not committed to a transitional government in Damascus, which the US has long stated would be a precondition for its involvement. Ban’s invitation enraged US diplomats and Syrian opposition groups, who were on the verge of abandoning the negotiations before they had started. But when Ban retracted his invitation Syria's opposition coalition immediately confirmed its participation in the talks, known as Geneva II.

The Syrian opposition coalition on Tuesday put out a statement saying the war crimes report "underlines the need for a transition to a Syria free of Assad", adding that: "In light of the new evidence, the Syrian coalition reiterates its call for the international community to exert all possible pressure on the regime to step aside and allow Syrians to realise their desire for a free country".

The statement said "the latest findings demonstrate that the level of industrialised killing the regime is inflicting on the country cannot be ignored" and called Russian and Iranian support for Assad "the main obstacle to finding a solution to the violence in Syria".

Oubai Shahbandar, an adviser to the Syrian opposition coalition, told Time that the release of the report could help the opposition. “This is unmistakable proof that the regime exists simply to maintain power,” Shahbandar said, “and will use whatever brutal means it has at its disposal to ensure Assad remains in power, which is why a political process is so crucial.”

Most of the photographic evidence featuring in the prosecutors' report comes from a defector from the regime, who said his job was "taking pictures of killed detainees" on behalf of Syria’s military police. The 31-page report was commissioned by a leading firm of London solicitors acting for Qatar, which backs Syria's rebel groups.

The inquiry team concluded there was “clear evidence, capable of being believed by a tribunal of fact in a court of law, of systematic torture and killing of detained persons by the agents of the Syrian government. It would support findings of crimes against humanity and could also support findings of war crimes against the current Syrian regime.”

One of the co-authors of the report, Sir Desmond de Silva QC, former chief prosecutor of the special court for Sierra Leone, told the Guardian that the evidence “documented industrial-scale killing”.

“This is a smoking gun of a kind we didn't have before,” he said. “It makes a very strong case indeed.”

That case will be pressed in the coming days by the US and Britain, who are leading efforts to bring Assad’s regime to an end.

The US official said: “As we have for over two years, and again today, we call on the Syrian government to grant immediate and unfettered access to all their detention facilities by international documentation bodies, including the UN commission of inquiry on Syria.

“We have long said that those responsible for atrocities in Syria must be held accountable for their gross violations of human rights.”

The report raises the stakes for the fragile peace talks, which have taken months to convene. The US secretary of state, John Kerry, is travelling to Switzerland for the start of the conference on Wednesday.

Ban's invitation to Iran and its subsequent withdrawal left US diplomats stunned. They pointed out that Iran had been pressed repeatedly to endorse the principles of an earlier summit, a requirement for participation in the talks.

“It’s been asked many times, and it’s always refused to do it,” said a second US official, based at the state department. “And so we just do not see how it can get an invitation.”