In February 1991, during the Gulf war, I found myself standing on the Saudi-Kuwait border, watching British infantrymen taking in Iraqi prisoners who were slipping across no-man's land in their twos and threes, desperate to surrender. The British were well prepared: there were large stockpiles of bottled water, tents and waiting vehicles.

Occasionally something went wrong, there were misunderstandings – especially at dawn or dusk – and people were hurt, but, by and large, what I witnessed was a humanitarian operation. The Iraqis were brought under shelter, rehydrated, given time to smoke, and then moved on. It was touching to see the care that some of the young British troops took when handling their enemies; conscripts who were bedraggled and terror-stricken after weeks of air bombardment.

Twelve years later I found myself with the British forces moving through Basra, clearing out pockets of opposition, sometimes dying in the process. At Saddam Hussein's palace on the banks of the Shatt al-Arab waterway, I turned a corner and came across a long row of Iraqi prisoners. They had hoods over their heads, their hands were plasti-cuffed behind their backs and they were kneeling bolt-upright in the sun.

An hour later I returned: the prisoners were still there, kneeling in the same positions.

Later that day a couple of army medics sidled up and confided that they had just been called upon to revive a prisoner who was being interrogated in a basement by British troops. There were a few prisoners down there, they explained, it smelt of vomit and shit and blood, and whatever was happening had caused one man to lose consciousness. The medics were clearly unsettled.

It is difficult, at times, for reporters in war zones to comprehend what is happening in front of them. A clearer view, the elusive "big picture", can be grasped sometimes only from a distance, or after the passage of time.

But even then it was clear to me that there must be a reason for the different approaches to prisoner-handling between 1991 and 2003. In time, I could see that the reason involved not just issues around the training and preparedness of British troops, but instructions and signals passing down through the chain of command.

Several years later, while reporting on counter-terrorism investigations in the UK and Pakistan, I kept coming across young British jihadists, some of them dangerous terrorists, who complained that they had suffered appalling torture when detained overseas, and that British intelligence officers had interrogated them in between each session, asking the very same questions that had just been put under torture. Some of these men bore the scars: incisions; missing fingernails; holes caused by drills; psychological trauma.

Officials and ministers always evaded questions about these allegations, responding each time with the same mantra: "The UK does not participate in, solicit, encourage or condone the use of torture." As time went on, with the allegations mounting and the mantra looking increasingly threadbare, official denial came to be accompanied by increased courtroom secrecy, and "investigations" by the intelligence and security committee were conducted, giving the intelligence agencies a clean bill of health.

Despite this, the truth began slowly to be prised into the open as the result of the efforts of a surprisingly small number of lawyers, parliamentarians and journalists. Gradually, evidence emerged that British Muslims were flown from Afghanistan to Guantánamo Bay after 9/11 because ministers decided they should be consigned there. Evidence emerged too that British intelligence agencies had permitted terrorism suspects to leave the UK, then suggested their detention by "liaison partners", before drawing up lists of questions to be put by these notorious torturers; and that the young MI5 and MI6 officers involved in these operations were following a carefully crafted policy document that had been drawn up by government lawyers and approved by ministers and senior officials.

Finally, a serendipitous moment amid the chaos of the Libyan revolution two years ago led to the discovery of documents showing how MI6 and Muammar Gaddafi's intelligence agents had co-operated in the abduction of two dissidents - and their wives and children - and that these men were flown to Tripoli where they were tortured.

On Thursday, what was left of the obfuscation and denial was swept aside by Sir Peter Gibson, a retired appeal court judge. He produced a 115-page report that made clear that he has seen a wealth of documentary evidence proving that British intelligence officers had been involved in the interrogation of detainees whom they knew were being mistreated; that showed some of those officers to have been in favour of hooding, stress positions, sleep deprivation and physical assaults; and that indicated that the agencies did, in fact, "condone, encourage or take advantage" of so-called rendition operations.

The report had sat on David Cameron's desk for 18 months. We now know that that is the time it took for Gibson and his inquiry partner Dame Janet Paraskeva to fight off attempts by MI6 and MI5 and their allies in the Cabinet Office to have it censored heavily before publication.

Gibson's inquiry was shut down by the government, but not before he had seen enough for him to pose 27 questions that need to be put to former ministers and intelligence chiefs.

It is unclear whether the public will ever see the answers to those questions. The coalition government has decided, after years of promising that the investigation would be independent, and led by a judge, that it should instead be handed to the intelligence and security committee. Human rights groups denounced this move as a prelude to another whitewash.

It may be that it is a little too late for a cover-up, however: a substantial amount of the truth appears to be out there.

We now know, for example, that on 10 January 2002, the decision that British Muslims captured in Afghanistan would be consigned to Guantánamo was set out in a telegram signed by Jack Straw.

We know that the following day, a Friday, MI5 and MI6 officers in the field received their interrogation guidance, informing them that they were under no obligation to report breaches of the Geneva conventions and permitting them to interrogate prisoners whom they knew were being tortured.

And we know that on the following Monday morning, Geoff Hoon, the defence secretary, mobilised the military's specialist interrogation unit, some of whose members were later involved in the mistreatment of detainees in Iraq.

So we know that a "gloves-off" decision was taken in Whitehall in early January 2002. We do not yet know who took that decision, or why.