An independent inquiry is to be held into allegations that British soldiers mutilated and murdered civilians in Iraq – and the government has been forced to admit that key documents had not been disclosed.

In a letter read out in court today, the defence secretary, Bob Ainsworth, said he "profoundly regrets" the failures to disclose relevant documents. Though he denied the allegations, he said he was now prepared to set up an inquiry under European human rights convention articles enshrining the right to life and prohibiting torture, or inhuman or degrading treatment.

The government made the concession after documents, which contradict claims made on oath by officials, were belatedly disclosed to the high court.

They reveal that ministers, possibly including Tony Blair, knew much more about the incident than they have admitted.

In a series of sharp exchanges, Lord Justice Scott Baker accused the Ministry of Defence of "a complete waste of time, at vast expense" for failing to disclose important evidence in hearings that have been going on for nearly two years.

He also lambasted the MoD for claiming gagging orders on information that had already been published. The judge described the ministry's behaviour as "extremely worrying".

The military police have consistently claimed an investigation into the allegations was unnecessary as the Iraqis had not complained about their treatment until much later.

Documents revealed to the court today showed that the Iraqi detainees had complained at the time to the Red Cross. They also show that ministers knew about them a week after the incident on 14 May 2004.

On that day British troops fought with insurgents in a fierce exchange known as "the battle of Danny Boy", a road checkpoint near Majar-al-Kabir in Maysan province. The next day the bodies of 20 Iraqis were returned to their families.

Six Iraqis have been seeking an inquiry into the allegation that some of the 20 were taken as prisoners to Camp Abu Naji, an army base, where they were interrogated and tortured before being killed.

The MoD says all 20 died "on the battlefield" and their bodies were taken to Abu Naji to be photographed, to see if one was an insurgent suspected of helping kill six military police the previous year. It says only nine live prisoners were taken to the camp, and all left alive. A subsequent military police investigations cleared UK forces of any wrongdoing.

Clive Lewis QC, for the MoD, said no decision had yet been made on what kind of inquiry would be set up. He suggested it could take the form of a police investigation, though not by the military police, or incorporated into a public inquiry – due to start later this month – into the death of Baha Mousa, the Basra hotel receptionist killed in British custody in 2003. The European convention states that an inquiry must be independent. Scott Baker warned the MoD he did not want an inquiry "just to disappear in the undergrowth".

Crucial documents disclosed yesterday show that Red Cross officials visited the British detention facility between 17 May and 19 May 2004, a few days after the incident. Though they praised some aspects of the detention centre, they referred to what they called "one major concern" – a Red Cross doctor said in some cases "injuries to wrists indicated excessive force".

He added that injuries to the face suggested they were inflicted when the detainees were being "held down" or "defenceless". The Red Cross asked for an investigation.

The documents show that the MoD hierarchy, including Adam Ingram, then armed forces minister, was told about the Red Cross concerns a few days later and that the military police had started an investigation. They include a draft letter from Ingram to 10 Downing Street passing on the information.

Rabinder Singh QC, for the Iraqis, told the high court yesterday: "What can't now be said is that the claimants [the Iraqis] did not make complaints and therefore there was no need for an investigation at the time."

It was impossible to exaggerate the seriousness of the matter, he said, noting that his clients had been called liars many times. He continued: "Somebody in the government must have known they had indeed done what they said in court."

He added: "Ministers were told the case was being investigated and it was not."

Singh singled out Colonel Dudley Giles, a senior military police officer and MoD witness. He had "not told the truth to this court, in evidence, he did not tell the truth" about when investigations into the allegations were first made, Singh said.

Scott Baker, sitting with Mr Justice Silber and Mr Justice Sweeney, showed their view about the MoD's attitude by awarding especially high "indemnity costs" to the Iraqis' lawyers.

The judge also agreed to stay the current judicial review hearings, but not until the MoD had said what kind of inquiry it proposed to set up.

Phil Shiner, solicitor for the Iraqis, said that for years they had been accused of lying "for propaganda reasons". He described it as "a momentous day".

The armed forces minister, Bill Rammell, said: "It is clear from the evidence that no one was murdered or ill-treated by British forces. It is also clear that British forces did not mutilate corpses on the battlefield, and there is independent expert testimony to support this."

He added: "However, these are serious allegations and we regret that we have failed to provide the court with timely and sufficient disclosure of information to enable them to determine the facts."