With efforts to force a public inquiry proving unsuccessful, a cache of at least 2,626 recordings remains mostly unseen

Not long after the 2003 invasion of Iraq, Lieutenant Colonel Nicholas Mercer, then the British military's most senior lawyer in the country, paid a visit to an interrogation centre that the army's intelligence corps had established near Basra.

He was appalled by what he saw.

Around 40 handcuffed Iraqis were being forced to squat on the ground with their hands cuffed high behind their backs. Dark blue hoods covered their heads and nearby a generator was running. Later, in a statement to the inquiry into the death of Baha Mousa, Mercer described what he saw as "repulsive" and added: "It's a bit like seeing a picture of Guantánamo Bay for the first time. It is quite a shock."

Mercer warned the interrogators that the use of hooding had been banned three decades earlier, and that it would impair the detainees' breathing. He was told, however, that the use of hoods and stress positions "was in accordance with British army doctrine on tactical questioning".

Mercer suspects the generators were intended to muffle the sound of whatever was happening inside the tents where interrogations took place.

He may well have been correct. A couple of years into the occupation, the intelligence corps began making video films of their interrogations in Iraq. At least 2,626 recordings were made, and one investigator who has watched them told the Guardian that the contained images that included "the good, the bad and the ugly".

Just a handful have been made public, during judicial review proceedings at the high court in London that lawyers brought in an attempt to force a public inquiry into the UK military's entire detention and interrogation regime during the five-and-a-half years of the occupation of the south-east of Iraq.

The films show detainees being threatened, intimidated, subjected to sensory deprivation and complaining of starvation.

One video shows a prisoner protesting that he is in pain. The interrogator shouts: "Good. I hope you die of cancer. I hope your kids die." At the end of that session the prisoner is ordered to replace the goggles and the interrogator shouts an order for a guard to take him away "for a little run". The earmuffs are replaced and the prisoner is dragged away by his thumbs.

Guards at the interrogation centre have told the Guardian that between interrogation sessions – and out of sight of the cameras – they were ordered to kick the prisoners and strike them with rifle butts while forcing them, blindfolded, around obstacle courses. They also say they were ordered to prevent prisoners going to sleep.

As the evidence mounted, the Ministry of Defence appeared increasingly nervous that the high court could conclude that the abuse was systemic, arising out of the training of troops and the instructions they had received, rather than the work of small numbers of "rogue" service personnel.

Four years ago – as part of an attempt to persuade the courts that they did not need to order a wide-ranging public inquiry – the MoD established the Iraq Historic Allegations Team (IHAT) to investigate complaints from detainees.

IHAT is currently investigating 93 allegations of mistreatment involving 179 people. Nobody has been charged with any serious offence. One man was fined after he was filmed beating an Iraqi.

Some MPs complain that the absence of any charges suggests that IHAT's £35.7m budget is a waste of public money; lawyers for former detainees say the failure to prosecute is evidence that a more effective investigation is needed.

IHAT is not just examining allegations of mistreatment of detainees. The MoD admitted to the Guardian in 2010 that at least seven Iraqi civilians had died in UK military custody, in addition to Baha Mousa.

IHAT is now examining 52 allegations of unlawful killing involving 63 deaths, some of them individuals who were in custody.

The deaths include one that the Guardian examined in detail in 2011 and 12. Tariq Sabri al-Fahdawi was one of 64 men detained at a roadblock shortly after the invasion and taken to a prison in western Iraq that was being run by US, British and Australian special forces. The existence of the prison was concealed from both military lawyers like Mercer, and from the Red Cross.

The men had been hooded, their thumbs were bound together, and they were being ferried to the prison aboard RAF Chinook helicopters. Fahdawi died aboard one of the helicopters, allegedly kicked to death by troops from the RAF regiment when he managed to slip out of the plastic ties around his thumbs.

The Guardian discovered that the RAF police who conducted the initial investigation waited more than a year before asking an RAF pathologist whether they should exhume Fahdawi's body. When the RAF pathologist said they should not – a decision that surprised a civilian pathologist – RAF prosecutors concluded that there was insufficient evidence to bring any charges.

Documents from the case obtained through the Freedom of Information Act and through leaks showed that the RAF police had failed even to establish the dead man's name.

Responsibility for service prosecutions has since been removed from the three armed forces and handed to the Service Prosecuting Authority, formed in 2009.

IHAT has re-investigated the death of Fahdawi, but it is unclear what conclusions it has reached. A report has been handed to the MoD, but the MoD says it will not be published.