A senior judge has been appointed to lead the public inquiry into the police’s controversial use of undercover officers to infiltrate political campaigns over more than 40 years.

The home secretary, Theresa May, told MPs on Thursday that Lord Justice Pitchford, who has sat in the court of appeal for five years, will head the investigation into the police’s failings and will have the power to compel witnesses to give evidence.

May ordered the inquiry last year after she was “profoundly shocked” to discover that a covert Scotland Yard unit, the special demonstration squad, had spied on the family of murdered teenager Stephen Lawrence.

It followed a series of revelations in the Guardian about sexual relationships with women including the fathering of children, the theft of dead children’s identities and the use of fake names in court.

Since 1968, undercover police officers have adopted fake personas and spent years infiltrating and disrupting hundreds of political groups. May said the inquiry was necessary as previous official investigations had “unearthed historical failings in undercover policing practices”.

“Undercover policing is an essential tactic in the fight against crime but to improve the public’s confidence in undercover work we must ensure there is no repeat of these failings in the future,” she added.

Pitchford, 67, became a barrister in 1969 and then a judge in 1987. Between 2000 and 2010, he sat in the Queen’s bench division of the high court.

Two years ago, Peter Francis, a whistleblower who went undercover to infiltrate anti-racist groups for four years, revealed that he had been part of a secret operation to spy on the Lawrences and their supporters, while they were campaigning to bring Stephen’s killers to justice.

In the last five years, the one-time SDS officer has disclosed many aspects of his former unit’s work.

He said he hoped that Pitchford “has the ability, clarity, desire, sense of purpose and indeed the honour to truly deliver a public inquiry that the Metropolitan police, this time round, cannot again deceive and manipulate as they unfortunately managed” to do in the 1999 inquiry, into their failure to investigate the Lawrence murder properly.

On Thursday, May said the inquiry would examine the operations of the SDS, which operated between 1968 and 2008; the national public order intelligence unit, which was set up to infiltrate political campaigns in 1999; and covert work by other police forces in England and Wales.

She said the remit of the inquiry – established under the 2005 Inquiries Act – is due to be published by July, following input from interested parties including those who were spied upon.

The Monitoring Group, a civil liberties body working with grieving families who were surveilled by undercover police, welcomed May’s announcement as an important step.

Duwayne Brooks, a friend of Stephen Lawrence who was also spied on, urged the home secretary to ensure that evidence did not go “missing or get tampered with”.

Mick Creedon, the Derbyshire Chief Constable who is leading the police’s internal investigation into the SDS, said the public inquiry “will help us with the work that is already underway to make sure that the unacceptable behaviour of some officers in the past never happens again”.

May said that other inquiries into unsafe convictions resulting from the undercover operations and possible criminal prosecutions “were much larger than initially envisaged” but should not delay the start of the inquiry by Pitchford.

May also said that a separate inquiry over the past year had found no evidence that the Home Office which funded the SDS between 1969 and 1989 had any knowledge of misconduct by the undercover unit.

