A police officer who spent four years living undercover in protest groups has revealed how he participated in an operation to spy on and attempt to "smear" the family of murdered teenager Stephen Lawrence, the friend who witnessed his fatal stabbing and campaigners angry at the failure to bring his killers to justice.

Peter Francis, a former undercover police officer turned whistleblower, said his superiors wanted him to find "dirt" that could be used against members of the Lawrence family, in the period shortly after Lawrence's racist murder in April 1993.

He also said senior officers deliberately chose to withhold his role spying on the Lawrence campaign from Sir William Macpherson, who headed a public inquiry to examine the police investigation into the death.

In extracts from a joint Guardian and Channel 4, police whistleblower Peter Francis reveals disturbing details about his undercover deployment The Guardian / Channel 4 Dispatches

Francis said he had come under "huge and constant pressure" from superiors to "hunt for disinformation" that might be used to undermine those arguing for a better investigation into the murder. He posed as an anti-racist activist in the mid-1990s in his search for intelligence.

"I had to get any information on what was happening in the Stephen Lawrence campaign," Francis said. "They wanted the campaign to stop. It was felt it was going to turn into an elephant.

"Throughout my deployment there was almost constant pressure on me personally to find out anything I could that would discredit these campaigns."

Francis also describes being involved in an ultimately failed effort to discredit Duwayne Brooks, a close friend of Lawrence who was with him on the night he was killed and the main witness to his murder. The former spy found evidence that led to Brooks being arrested and charged in October 1993, before the case was thrown out by a judge.

Peter Francis, the former undercover police officer turned whistleblower. Photograph: Graham Turner for the Guardian

The disclosures, revealed in a book about undercover policing published this week, and in a joint investigation by the Guardian and Channel 4's Dispatches being broadcast on Monday, will reignite the controversy over covert policing of activist groups.

Lawrence's mother, Doreen, said the revelations were the most surprising thing she had learned about the long-running police investigation into her son's murder: "Out of all the things I've found out over the years, this certainly has topped it."

She added: "Nothing can justify the whole thing about trying to discredit the family and people around us."

In a statement, the Metropolitan police said it recognised the seriousness of the allegations – and acknowledged their impact. A spokesman said the claims would "bring particular upset" to the Lawrence family and added: "We share their concerns."

Jack Straw, the former home secretary who in 1997 ordered the inquiry that led to the 1999 Macpherson report, said: "I'm profoundly shocked by this and by what amounts to a misuse of police time and money and entirely the wrong priorities." Straw is considering personally referring the case to the Independent Police Complaints Commission.

Francis was a member of a controversial covert unit known as the Special Demonstration Squad (SDS). A two-year investigation by the Guardian has already revealed how undercover operatives routinely adopted the identities of dead children and formed long-term sexual relationships with people they were spying on.

The past practices of undercover police officers are the subject of what the Met described as "a thorough review and investigation" called Operation Herne, which is being overseen by Derbyshire's chief constable, Mick Creedon.

A spokesman said: "Operation Herne is a live investigation, four strands of which are being supervised by the Independent Police Complaints Commission, and it would be inappropriate to pre-judge its findings."

Francis has decided to reveal his true identity so he can openly call for a public inquiry into undercover policing of protest. "There are many things that I've seen that have been morally wrong, morally reprehensible," he said. "Should we, as police officers, have the power to basically undermine political campaigns? I think that the clear answer to that is no."

Francis has been co-operating with the Guardian as a confidential source since 2011, using his undercover alias Pete Black. He assumed the undercover persona between 1993 and 1997, infiltrating a group named Youth Against Racism in Europe. He said he was one of four undercover officers who were also required to feed back intelligence about the campaigns for justice over the death of Lawrence.

Francis said senior officers were afraid that anger at the failure to investigate the teenager's racist killing would spiral into disorder on the streets, and had "visions of Rodney King", whose beating at the hands of police led to the 1992 LA riots.

Francis monitored a number of "black justice" campaigns, involving relatives of mostly black men who had died in suspicious circumstances in police custody.

However, he said that his supervising officers were most interested in whatever information he could gather about the large number of groups campaigning over the death of Lawrence.

Although Francis never met the Lawrence family, who distanced themselves from political groups, he said he passed back "hearsay" about them to his superiors. He said they wanted information that could be used to undermine the campaign.

One operation Francis participated in involved coming up with evidence purporting to show Brooks involved in violent disorder. Francis said he and another undercover police officer trawled through hours of footage from a May 1993 demonstration, searching for evidence that would incriminate Brooks.

Police succeeded in having Brooks arrested and charged with criminal damage, but the case was thrown out by a judge as an abuse of the legal process. Francis said the prosecution of Brooks was part of a wider drive to damage the growing movement around Lawrence's death: "We were trying to stop the campaign in its tracks."

Doreen Lawrence said that in 1993 she was always baffled about why family liaison officers were recording the identities of everyone entering and leaving their household. She said the family had always suspected police had been gathering evidence about her visitors to discredit the family.

"We've talked about that several times but we never had any concrete [evidence]," she said.

There is no suggestion that the family liaison officers knew the purpose of the information they collected.

Francis claims that the purpose of monitoring people visiting the Lawrence family home was in order "to be able to formulate intelligence on who was going into the house with regards to which part of the political spectrum, if any, they were actually in". The former policeman added: "It would determine maybe which way the campaign's likely to go."

In 1997, Francis argued that his undercover operation should be disclosed to Macpherson, who was overseeing the public inquiry into the Met's handling of the murder. "I was convinced the SDS should come clean," he said.

However his superiors decided not to pass the information on to the inquiry, he said. He said he was told there would be "battling on the streets" if the public ever found out about his undercover operation.

Straw said that neither he nor Macpherson were informed about the undercover operations. "I should have been told of anything that was current, post the election of Tony Blair's government in early May 1997," he said.

"But much more importantly, [the] Macpherson inquiry should have been told, and also should have been given access to the results of this long-running and rather expensive undercover operation."