More than half of the country’s undercover police officers are not on the official database supposed to record them, according to a damning report into undercover policing published today.

This comes amid mounting concern over the activities of undercover police, sparked by revelations in 2011 that Metropolitan Police officer Mark Kennedy had spied on environmental protesters and had affairs with a number of them. Allegations of undercover officers using birth certificates of dead babies are among other scandals which have since come to light. And in March 2014 Home Secretary Theresa May announced a public inquiry into undercover police officers after revelations that the family of Stephen Lawrence had been spied on.

In the first ever inspection of undercover policing in England and Wales, Her Majesty’s Inspectorate of Constabulary (HMIC) accuses senior officers of letting down the undercover officers under their command. It reveals how there are 1,229 undercover officers across the country. But the national undercover index only has 568 officers – less than half the true number. This discrepancy is “entirely unacceptable” and “renders the database unsuitable to the task for which it was created,” says the report.

In general, those who go undercover “carry out their roles professionally and with great courage”. But “material weaknesses” among senior officers have resulted in “unacceptable inconsistencies in areas of critical importance, such as the provision of training and operating procedures that should assist officers to carry out their role, confident that they are acting lawfully, effectively and to the same high standards”.

The report, based on an inspection of 43 forces and other organisations such as the National Crime Agency, also condemns the “culture of secrecy” and “closed nature” which acts as a barrier to scrutiny of undercover policing. It calls for officers to not spend any longer than 10 years in undercover work, after inspectors found one officer who has served undercover for more than 20 years.

And undercover officers should face random drug testing, says the report: “We were concerned to be told by many undercover officers that they had not once been subjected to a random drugs test, despite the fact that the officers themselves saw such testing as an important protection for them.”

Metropolitan Police officer Mark Kennedy was revealed in 2011 to have had affairs with protesters, sparking calls for scrutiny of undercover work (Channel 4)

A “lack of effective and coordinated direction” by the National Undercover Working Group – which is responsible for the strategy and policy for undercover policing – is “unacceptable”. And a "root and branch" reform of the working group is needed. Among the other recommendations in the report is a call for “psychological support for all undercover officers”.

Many senior officers who authorise undercover operations have no training or experience in this area, says the report. And only 25 forces have a “dedicated undercover online capability” which means “a number of police forces are missing or avoiding the opportunity to deploy and effective and efficient tactic to protect their communities from a modern threat”.

There are also concerns over the ways in which operations are authorised. In some instances sections from previous forms had been copied and pasted when applying for authorisation for new undercover operations. In one force, this resulted in an undercover online investigator into the sexual exploitation of children “inadvertently” given permission to purchase firearms.

Video: 'Undercover practices need to be changed'

HM Inspector Stephen Otter, who led the inspection, said: “We were concerned by a generally poor level of knowledge and lack of expertise of those senior leaders who authorise the use of undercover officers. there was an unacceptable variation in the quality of the written authorisations - often with insufficient detail provided.”

Ms May, who commissioned the HMIC inspection, said: "More must be done to improve the public's confidence in undercover work. this Government has already introduced a number of measures to increase oversight and transparency in undercover policing, including raising the authorisation level for undercover officers.”

The College of Policing and relevant national policing leads have been asked to produce an action plan and timetable for carrying out the recommendations in the report, she added.

Responding to the HMIC report, Deputy Chief Constable Jon Boutcher, the national policing lead for undercover policing, said: “Police leaders have a responsibility to make sure that these operations are properly authorised, managed and overseen. We support HMIC’s recommendations and are already putting them into place.”