The money came pouring in. Jonathan Rees worked from a dingy office in south London. He lived in a cramped flat upstairs. He was divorced, overweight and foul-mouthed but his business was golden: he traded information. His sources may have been corrupt. His actions may have been illegal. But the money kept coming – from one golden source in particular. As Rees himself put it: "No one pays like the News of the World do."

There was only one problem with Rees's lucrative business. He had caught the eye of Scotland Yard's anti-corruption command who strongly suspected that he was paying bribes to various serving officers and, with great care and some skill, they had managed to place a covert listening device inside his office.

It was that bug which recorded him gloating about the pay he received from the News of the World. It also recorded the vivid detail of an empire of corruption, run with casual ease by Rees and his business partner, Sid Fillery – and liberally greased with cash from the News of the World and other Fleet Street titles. The News of the World alone was paying him more than £150,000 a year.

The listening device was placed in Rees's office in mid-April 1999. It did its job for only six months. In that short time, it provided one highly revealing chapter in a long tale of promiscuous criminality. Further chapters were provided by three other private investigators, all of whom worked separately for the News of the World, all of whom finally ended up in court, all of whom were publicly linked with illegal news-gathering.

Over the following years, the Guardian published a lengthy exposé of Rees's involvement with corrupt police and the procurement of confidential information for the News of the World; the Sunday tabloid's assistant editor is believed to have been arrested and accused of paying bribes to police and other key workers, although he was never charged; the paper was named in a London court as the paymaster for the purchase of information from the police national computer; Rees was jailed for a conspiracy to frame an innocent woman and then accused of conspiracy to murder.

And yet the man who became the prime minister's media adviser, Andy Coulson, has always maintained in evidence to parliament and on oath in court that he knew nothing of any illegal activity during the seven years he spent at the top of the News of the World. The entire story unfolded without ever catching his eye. In the same way, the prime minister and his deputy were happy to appoint Coulson last May to oversee the communication between the British government and its people, even though they were already fully aware of all the essential facts.

It begins with the bug. It is commonplace for journalists to interview police officers, but the listening device recorded Rees routinely paying cash directly or indirectly to serving officers, a serious criminal offence. By April 1999, Rees had been working for Fleet Street for several years, and he had created a vibrant network of corrupt sources.

The bug recorded the sound of Detective Constable Tom Kingston from the south-east regional crime squad collecting cash for himself and for his mate who was an intelligence officer involved in the protection of the royal family and other VIPs. DC Kingston sold Jonathan Rees a Special Branch report disclosing police knowledge of an Albanian crime gang in London, Police Gazette bulletins which listed suspects who were wanted for arrest, and threat assessments in relation to the terrorist targets his mate was supposed to be protecting. Rees sold them to newspapers – primarily the News of the World, the Sunday Mirror and the Daily Mirror.

DC Kingston eventually ended up in prison for selling a huge quantity of amphetamine which he had stolen from a dealer. But Rees had other links to other corrupt officers. His partner, Sid Fillery, was a former officer who had connections all over the force. The bug recorded their relationship with Duncan Hanrahan and Martin King, who had left the Metropolitan police to work as private investigators and who were similarly well connected until both were jailed in relation to police corruption. Hanrahan also admitted conspiring to rob a courier of £1m at Heathrow airport.

Some of what they sold was tittle-tattle: a disparaging remark made by Tony Blair about John Prescott within earshot of a bent officer; gossip about the sex lives of Buckingham Palace servants. But some of it was highly sensitive. When one of Britain's most notorious criminals, Kenneth Noye, was finally arrested, Rees bought and sold details of the secret intelligence which had led to his capture as well as the precise time and route by which he would be driven from prison to court. When the TV journalist Jill Dando was murdered on her doorstep, Rees procured a police source so that he could sell live details of the investigation.

And the corruption did not stop with the police. The listening device caught Rees boasting that he was in touch with: two former police officers working for Customs and Excise who would accept bribes; a corrupt VAT inspector who had access to business records; and two corrupt bank employees who would hand over details of targets' accounts. (One of them had the first name Robert and was wittily referred to as Rob the Bank. The other was simply Fat Bob.)

Beyond that, Rees regularly hired two specialist "blaggers" who tricked their way into phone company records to obtain names and home addresses of subscribers and also itemised phone bills, highly valued on Fleet Street as a list of potential sources. One was a Londoner named Shaun who routinely provided lists of phone numbers called by Rees's targets. The other was John Gunning, a private investigator who has since been convicted of blagging the private information of subscribers from British Telecom's database.

Rees's relationship with journalists was a two-way street. An executive from the News of the World developed a corrupt source in the Passport Office who could provide home addresses, personal details and photographs of anybody who applied for a passport. Rees was paying the executive £100 a time for information from the source (although the executive was passing the source only £25 a time).

One person who is familiar with Rees's operations claims that he or one of his associates started using Trojan Horse software, which allowed them to email a target's computer and copy the contents of its hard disk. This source claims that they used this tactic when they were hired by the News of the World to gather background on Freddy Scapaticci, a former IRA man who had been exposed as an MI6 informer codenamed Stakeknife.

Two other sources claim that Rees was commissioning burglaries. One is a private investigator who was told directly by Rees's network that they had broken into targets' home on behalf of a Fleet Street newspaper. The other is a lawyer who claims to have evidence that a high-profile client was the target of an attempted burglary by Rees's associates in search of embarrassing information. There is no independent confirmation of this.

The bug betrayed the sheer speed and ease with which Rees was able to penetrate the flimsy fence of privacy that shields the vast reservoir of personal information now held on the databases controlled by the police and the DVLA, the phone companies and banks. On one occasion, he was asked to find out about the owner of a Porsche. Armed with the registration number, it took him a grand total of 34 minutes to come up with the owner's name and home address from the DVLA and his criminal record from the police computer.

When the Daily Mirror wanted the private mortgage details of all the governors of the Bank of England, Rees delivered.

When the Sunday Mirror wanted to get inside the bank accounts of Prince Edward and the Countess of Wessex, it was equally easy, as the bug recorded:

Reporter: "Do you remember a couple of months ago, you got me some details on Edward's business and Sophie's business and how well they were doing?"

Rees: "Yeah."

Reporter: "And you did a check on Sophie's bank account."

Rees: "Yeah."

Reporter: "Is it possible to do that again? I'm not exactly sure what they're after but they seem to be under the impression that, you know, she was in the paper the other day for appearing in Hello magazine. They think she's had some kind of payment off them."

Rees: "What? Off Hello?"

Reporter: "Um, yeah."

Rees: "… find out how much."

Reporter: "Well, we just want to see if there's been any change to her bank account. "

This would be a breach of the Data Protection Act unless the courts held there was a clear public interest in establishing the health of the countess's business or her deal with Hello magazine. The payment of bribes would be a criminal offence regardless of any public interest. Rees made no secret of his criminality. At one point the police bug caught Rees telling a Daily Mirror journalist that they must be careful what they wrote down "because what we're doing is illegal, isn't it? I don't want people coming in and nicking us for a criminal offence, you know."

But Rees did get nicked – and for a serious criminal offence. The listening device caught him being hired by a man who was getting divorced and wanted to stop his wife getting custody of their children. Rees came up with a plan. Aided and abetted by yet another corrupt police officer, DC Austin Warnes, he arranged to plant cocaine in the car of the unsuspecting woman, so that she could be charged, convicted and smeared as an unreliable parent.

In order to stop that plot, in September 1999, Scotland Yard raided Rees and charged him with conspiracy to pervert the course of justice. Fifteen months later, he was taken off Fleet Street's payroll when he was sentenced to six years in prison, increased to seven years on appeal. DC Warnes was sentenced to four years.

And none of this was secret. Apart from the case itself, which was held in open court, the Guardian two years later, in September 2002, ran a three-part series on invasion of privacy and devoted some 3,000 words to a detailed account of Rees's dealings with corrupt police officers and of his use generally of illegal methods to acquire information for the News of the World and other papers.

Based on an authorised briefing by Scotland Yard, the Guardian story made repeated references to the News of the World's involvement and quoted an internal police report to the effect that Rees and his network were involved in the long-term penetration of police intelligence and that "their thirst for knowledge is driven by profit to be accrued from the media". The Crown Prosecution Service found that there was no evidence that the reporters involved knew that Rees was acquiring the material by corrupt means.

A year later, in August 2003, Sid Fillery, who was still running the agency and working for Fleet Street, also got himself arrested and charged with 15 counts of making indecent images of children and one count of possessing indecent images. This was reported in national media. He was later convicted.

All of this extraordinary and well-publicised activity around the News of the World nevertheless apparently escaped the attention of Andy Coulson, even though he had been hired early in 2000 to be deputy editor of the paper under his close friend, Rebekah Wade. And Jonathan Rees was not the only private investigator who was routinely breaking the law for the News of the World without Coulson knowing anything at all about it.

All through the late 1990s, the paper had been hiring an investigator called John Boyall, who, among other services, specialised in acquiring information from confidential databases. He had a wiry young man working as his assistant, named Glenn Mulcaire. In the autumn of 2001, John Boyall fell out with the News of the World's assistant editor, Greg Miskiw, who had been responsible for handling him. Miskiw replaced him by poaching Glenn Mulcaire and giving him a full-time contract.

Mulcaire – as the world now knows – proceeded to hack the voicemail messages of public figures. Journalists who worked at the paper say that reporters were routinely and openly hacking voicemail themselves. Some of them were doing it for stories. Some of them were doing it out of idle curiosity. One reporter, for example, was obsessed with Liz Hurley and routinely listened to her messages and to those of her hairdresser or driver or anybody else connected with her. Mulcaire personally hacked voicemail for special projects or when reporters found they could not do it themselves. He also blagged information from phone companies and banks.

Two reporters from that era have spoken on the record. Paul McMullan, who was a deputy features editor, told the Guardian that he had personally commissioned several hundred illegal acts from private investigators and that his deputy editor, Andy Coulson, must have known about it. Sean Hoare, a legendary showbiz writer, told the New York Times, that Coulson, who was then a close friend, had actively encouraged him to hack voicemail and had listened to messages with him. Coulson denies all this.

In January 2003, Coulson replaced Rebekah Brooks as editor of the News of the World. Two months later, in March 2003, they gave evidence together to the House of Commons media select committee and, in answer to a direct question, Rebekah Brooks declared: "We have paid the police for information in the past."

Asked if she would do so again in the future, her answer was pre-empted by Coulson. "We operate within the law and, if there is a clear public interest, then we will," he told the committee. It was pointed out to Coulson that it was always illegal to pay police officers, regardless of public interest. Coulson suggested he had been talking about the use of subterfuge.

Coulson's ignorance, however, was severely tested by an explosive avalanche of disclosure which began that same month when the Information Commissioner's Office raided the home in New Milton, Hants ,of a private investigator named Steve Whittamore and seized a mass of paperwork which turned out to be a detailed record of more than 13,000 requests from newspapers and magazines for Whittamore to obtain confidential information, many of them potentially in breach of the law. Several staff from the Guardian's sister paper, the Observer, were among Whittamore's customers.

In a blue exercise book, Whittamore recorded all his transactions with the News of the World. He identified 27 different journalists as commissioning his work – well over half of the news and feature writers on the paper, spending tens of thousands of pounds. Greg Miskiw alone was recorded as making 90 requests. Whittamore's invoices, submitted for payment by News International's accounts department, sometimes made explicit reference to obtaining a target's details from their phone number or their vehicle registration.

Whittamore had been running a network of "blaggers" – a hell's angel on the south coast who specialised in posing as a British Telecom engineer to trick their call centres into handing over confidential data on subscribers; two employees of the DVLA who sold the details of car owners; an old friend called John Gunning, who specialised in extracting information from phone companies (and who had been one of Jonathan Rees's two specialist blaggers); and John Boyall – the private investigator who had been working directly for the News of the World until he fell out with them in 2001. And Boyall soon became a pivotal figure for the investigation.

Studying Whittamore's paperwork, the Information Commissioner realised that he was also able to buy information from the police national computer. They contacted Scotland Yard, whose anti-corruption command set up Operation Glade and uncovered a chain of links. A newspaper would ask Whittamore for police data; Whittamore would ask Boyall; Boyall would ask a recently retired officer called Alan King; and King would obtain the information from a civilian police worker in Wandsworth, called Paul Marshall, who invented phone calls from members of the public to justify accessing the police national computer.

This led to the arrest of Boyall. Worse still, from the News of the World's point of view, it is also believed to have led to the arrest of the assistant editor who had dealt with Boyall, Greg Miskiw. The Guardian has put it to Miskiw that he was arrested and questioned at Colindale police station in north London, in the presence of a News International solicitor, about allegations that he had paid cash through Boyall to obtain information from the police computer; that he had also authorised the payment of cash bribes to other sources including the employees of mobile phone companies; and that during this interview with police, he exercised his right to make no comment. Miskiw gave no response to these questions from the Guardian.

Miskiw was not charged with any offence. On April 2005, Steve Whittamore and John Boyall – both of whom had worked directly for the News of the World – appeared at Blackfriars crown court with the two men who had given them access to the police computer. All four pleaded guilty to procuring confidential police data to sell to newspapers. The News of the World was named in court as one of their buyers. The case was reported in national newspapers.

2005 continued to throw up incidents which Coulson failed to notice. Crucially, it was now that Jonathan Rees found his way back to the News of the World. In spite of his prison sentence for attempting to frame an innocent mother and in spite of the Guardian's long exposé of his illegal activities on behalf of Coulson's paper, Rees once again was plying his trade and being paid for it from Coulson's budget. He continued to do so until April 2008 when he was charged with conspiring to murder his former business partner, Daniel Morgan, in 1987, a charge which was finally dismissed yesterday at the Old Bailey.

Beyond that, we now know that during 2005, Glenn Mulcaire was being commissioned regularly to hack voicemail. Evidence which has recently been disclosed in court cases suggests that at least three senior journalists were involved in the commissioning: Greg Miskiw, Ian Edmondson and Clive Goodman. Those three men have one thing in common: all of them worked as news editor of the News of the World, raising the possibility that it was part of their job description to commission work from Mulcaire, the only private investigator who worked on a full-time contract for the paper.

If Andy Coulson had failed to notice any of this consistently illegal activity in his newsroom involving Rees, Whittamore, Boyall and Mulcaire; if none of the reporters who worked with these investigators and/or hacked the voicemail of their targets ever mentioned anything to him; if nobody told him there was police activity around his assistant editor, Greg Miskiw; if he failed to ask why the editorial budget had poured hundreds of thousands of pounds into these investigators; if he failed to read any of the news reports that linked his newspaper to Jonathan Rees's corruption or to Whittamore and Boyall's network of blaggers: finally, in the late summer of 2006, the reality caught his eye.

Scotland Yard arrested Clive Goodman, a former news editor then working as royal correspondent, and Glenn Mulcaire. When they appeared in court in January 2007, they pleaded guilty to hacking the messages of eight public figures and were jailed. Coulson resigned insisting that he must take responsibility even though he had never known anything about it.