A national police unit that uses undercover officers to spy on political groups is currently monitoring almost 9,000 people it has deemed "domestic extremists".

The National Domestic Extremism Unit is using surveillance techniques to monitor campaigners who are listed on the secret database, details of which have been disclosed to the Guardian after a freedom of information request.

A total of 8,931 individuals "have their own record" on a database kept by the unit, for which the Metropolitan police is the lead force. It currently uses surveillance techniques, including undercover police, paid informants and intercepts, against political campaigners from across the spectrum.

Senior officers familiar with the workings of the unit have indicated to the Guardian that many of the campaigners listed on the database have no criminal record.

As as Scotland Yard was battling to contain the fallout over the activities of a former undercover police officer who was asked to dig for "dirt" that would undermine the Stephen Lawrence campaign, evidence emerged that the main witness to his murder was also targeted.

Sources indicated that the Met secretly bugged meetings with Duwayne Brooks and his solicitor. The surveillance operation was understood to have been authorised by a "senior officer" in around 1999 or 2000.

At least two meetings are believed to have been covertly recorded, one of them at the offices of Brooks's solicitor, Jane Deighton. She told the BBC, which first reported the story, that if true the operation was "scandalous".

Last night it emerged that Stephen Lawrence's mother Doreen is to meet with the home secretary on Thursday morning.

The Met commissioner, Bernard Hogan-Howe, is resisting calls for an independent inquiry into the latest revelations. His force said it recognised the "huge seriousness" of the fresh claims about the surveillance of Brooks, who is now a Lib Dem councillor in South London, and would investigate them internally.

Former undercover officer Peter Francis had previously revealed he was involved in an ultimately failed operation to discredit Brooks, seeking information that was used to bring an unsuccessful prosecution for criminal damage in 1993, a few months after Lawrence died. Francis's full story is told in a book about several undercover operations, published this week.

Francis's unit, the Special Demonstration Squad (SDS), was disbanded in 2008, but later replaced with the National Domestic Extremism Unit.

The extremism unit monitors the full range of activists: from far-right activists in the English Defence League through to animal rights protesters, anti-capitalists and anti-war demonstrators.

In recent years the unit is known to have focused its resources on spying on environmental campaigners, particularly those engaged in direct action and civil disobedience to protest against climate change.

A small number of activists have obtained excerpts from their file in the extremism unit's database. They include an 88-year-old campaigner, John Catt, who won a landmark lawsuit against the Met three months ago. Three court-of-appeal judges ruled the Met had unlawfully retained details of the pensioner's presence at more than 55 protests. Details were logged about slogans on his banner and whether he was clean-shaven.

Another activist, Guy Taylor, 46, who campaigns against capitalism, discovered that he was spied on while attending Glastonbury festival – which is known to have been frequented by a number of police spies in recent decades. Taylor has one conviction for spray-painting a slogan in 1991.

He and Catt are among the thousands of activists who have been categorised as domestic extremists on the unit's files. The Met previously used the term "subversives" to describe citizens with radical political views whom it was spying on.

On Tuesday, Francis said in a Guardian webchat that those targeted by Special Branch in the past included the former home secretary, Jack Straw, once a student union activist.

"I read Mr Straw's rather large file," he said. "It will be a pink file with his individual 'RF' (Registry File) number. The same for [MPs] Diane Abbott and Jeremy Corbyn – and Imran Khan, the lawyer for the Stephen Lawrence family. The human rights solicitor firm Bindmans also had its own dedicated file."

Francis also said a low point of his deployment as an anti-racist campaigner in the 1990s came when he undermined the campaign of a family who wanted justice over the death of a boxing instructor who was struck on the head by a police baton.

He said he had infiltrated the family-led campaign for justice over the death ofBrian Douglas, a 33-year-old who died after he was hit on the head with a police baton in 1995 when he was stopped for driving erratically.

"The lowest point I reached morally was when I was standing outside Kennington police station for the Brian Douglas justice campaign in May 1995. It was a candlelit vigil and his relatives were all there," he said.

"By me passing on all the campaign information – everything that the family was planning and organising through Youth Against Racism in Europe – I felt I was virtually reducing their chances of ever receiving any form of justice to zero. To this day, I personally feel that family has never had the justice they deserved."

Francis said he had "no faith" in the two existing inquires that the home secretary, Theresa May, has said will look into his allegations. One is an inquiry by a barrister into previously-known allegations of corruption in the investigation in the Lawrence murder, while the second, Operation Herne, is an internal Met police review being led by the chief constable of Derbyshire police.

"Only a judicial-led or public inquiry – not just into the Stephen Lawrence allegations, but into the wider controversy – has any chance of ever establishing the truth," he said.