The prime minister's media adviser, Andy Coulson, freely discussed the use of unlawful news-gathering techniques while editor of the News of the World and "actively encouraged" a named reporter to engage in the illegal interception of voicemail messages, according to allegations published by the New York Times.

Coulson, who resigned as editor of the News of the World in January 2007 after its royal correspondent was jailed for intercepting voicemail messages, has always insisted that he had no knowledge of illegal activity when he edited the paper or at any time as a journalist. He told a Commons select committee last year: "I have never had any involvement in it at all."

The New York Times website published a trail to a story due to appear in its Sunday magazine. It made detailed allegations likely to bring intense new pressure on Coulson and the Metropolitan police force, which stands accused of favouring Rupert Murdoch's newspaper group by cutting short its investigation, withholding crucial evidence from prosecutors and failing to inform victims of the newspaper's crimes against them. Coulson declined to comment on the allegations. The News of the World and Scotland Yard have denied all the charges.

Coulson resigned after the imprisonment of his royal reporter, Clive Goodman, and a private investigator, Glenn Mulcaire, for "hacking" into the voicemail messages of eight public figures. When the Guardian revealed last year that the scandal involved other journalists at the paper and numerous other victims, Coulson said he had nothing to add to earlier denials of involvement, and the Conservative leader stood by him. David Cameron said: "I believe in giving people a second chance."

The New York Times, which has had an investigative team at work on the story since March, is citing two former News of the World journalists who specifically claim that Coulson was directly aware of his reporters' use of illegal techniques.

An unnamed former editor is quoted as claiming that Coulson talked freely about illegal news-gathering techniques, including phone-hacking, and that he personally had been at "dozens, if not hundreds" of meetings with Coulson where the subject came up. "The editor added that when Coulson would ask where a story came from, editors would reply 'We've pulled the phone records' or 'I've listened to the phone messages'."

In addition, Sean Hoare, a former reporter who used to be a close friend of Coulson, is quoted as saying that when he worked with Coulson at the Sun, he personally played recordings of hacked voicemail messages for him and that later, when he worked for Coulson at the News of the World, he "continued to inform Coulson of his pursuits. Coulson 'actively encouraged me to do it', Hoare said".

Hoare, who was sacked from the paper at a time when he had drink and drug problems, says he personally listened to the voicemail messages of celebrities such as David and Victoria Beckham and that he has spoken out now because he believes it was unfair for Goodman to get all the blame.

Coulson told the Commons media committee last year that he had never even heard Mulcaire's name and that Goodman had been the only reporter involved: "I am absolutely sure that Clive's case was a very unfortunate rogue case."

The New York Times claims to have spoken to a dozen former News of the World reporters and editors who say that phone-hacking was "pervasive" in Coulson's newsroom. "Everyone knew," according to an unnamed senior reporter. "The office cat knew." Most former reporters are unnamed, but Sharon Marshall is named as having witnessed hacking when working under Coulson from 2002-04. "It was an industry-wide thing," she said.

The paper says that Coulson ran a highly competitive newsroom "with single-minded imperiousness". Former News of the World journalists claim that there was a "do whatever it takes" mentality and that reporters were told to "get the story, no matter what". "They described a frantic, sometimes degrading atmosphere in which some reporters openly pursued hacking or other improper tactics to satisfy demanding editors," according to the New York Times.

The paper gives a specific example of the involvement of an editorial executive: "Matt Driscoll, a former sports reporter, recalled chasing a story about the soccer star Rio Ferdinand. Ferdinand claimed he had inadvertently turned off his phone and missed a message alerting him to a drug test. Driscoll had hit a dead end, he said, when an editor showed up at his desk with the player's private phone records." Driscoll was later dismissed and awarded £800,000 by a tribunal, which found that he had been bullied by Coulson.

Bill Akass, managing editor of the News of the World, dismissed the New York Times claims as "unsubstantiated". He said: "We reject absolutely any suggestion or assertion that the activities of Clive Goodman and Glenn Mulcaire, at the time of their arrest, were part of a culture of wrongdoing at the News of the World and were specifically sanctioned or accepted at a senior level in the newspaper."

The New York Times goes on to quote unnamed sources from the Met suggesting that its inquiry into the phone hacking was hampered by a desire to avoid upsetting Britain's biggest selling newspaper: "Several investigators said in interviews that Scotland Yard was reluctant to conduct a wider inquiry in part because of its close relationship with the News of the World."

After a raid on Goodman's desk in August 2006, according to the New York Times, "several detectives said they began feeling internal pressure. One senior investigator said he was approached by someone from the department's press office, who was waving his arms in the air, saying 'wait a minute, let's talk about this'."

The investigator, who has since left Scotland Yard, added that the press officer stressed the department's "long-term relationship with News International". The investigator recalled furiously responding: "There's illegality here, and we'll pursue it like we do any other case." Scotland Yard says that operational decisions are made by police, not by press officers.

Former journalists told the New York Times that when Scotland Yard raided Goodman's desk, two senior journalists "stuffed reams of documents into trash bags and hauled them away". Police did not interview any other reporter or editor apart from Goodman. The material seized from Goodman and Mulcaire included paperwork which potentially implicated three named journalists. None was interviewed and, as the Guardian disclosed last year, the police failed to pass key paperwork to the Crown Prosecution Service.

The New York Times quotes an unnamed former senior prosecutor who was "stunned to discover later that the police had not shared everything. 'I would have said we need to see how far this goes' and 'whether we have a serious problem of criminality on this news desk', said the former prosecutor."

When the case came to court, police identified eight victims of the hacking. However, the New York Times claims that the officer responsible for the inquiry, the then assistant commissioner Andy Hayman, had been shown a "target list" of names and numbers taken from Mulcaire's home which ran to eight or 10 pages and which "read like a British society directory".

The Met told prosecutors that it would approach all known victims, but failed to do so. One who was approached, the then Respect MP George Galloway, told the New York Times that police warned him that his voicemail had been intercepted but refused to tell him who was responsible.

Scotland Yard denies cutting short its inquiry or being influenced by its relationship with the News of the World. The Press Complaints Commission was criticised after two inquiries into the affair failed to find evidence of wrongdoing other than that originally presented by police.

After revelations in the Guardian, the Commons media select committee held a second inquiry into the affair last year. Its report expressed concern "at the readiness of all of those involved – News International, the police and the PCC – to leave Mr Goodman as the sole scapegoat without carrying out a full investigation".

Coulson said tonight: "I absolutely deny these allegations."