The security services may soon be able to scan sensitive data from a significant proportion of the British population for signs of terrorism, an official report reveals.

David Anderson QC examined reforms put in place by MI5 and counter-terrorism policing after the 2017 attacks in Britain. His report for the government on progress so far reveals that counter-terrorism investigators may soon have powerful artificial intelligence and behavioural analysis tools, similar to those used by Facebook, to assess masses of data.

Anderson warns that greater use of artificial intelligence and behavioural analytics needs to be ethical. Areas where it could be used include buying a flight, financial data and the hiring of vehicles. In three of the 2017 attacks, at Westminster Bridge, London Bridge and Finsbury Park, terrorists hired vehicles to use as weapons.

In his report on Tuesday, Anderson says: “More controversial, however, should increasing automation render it cost-effective, would be the use of such general indicators across the population as a whole, or significant portions of it, with a view to identifying possible future threats.”

The barrister warns of a possible row and damage to trust in state agencies, as when the Guardian revealed the evidence from whistleblower Edward Snowden about wide-ranging surveillance powers.

Anderson wrote: “As UKIC [the United Kingdom intelligence community] came to acknowledge in the wake of the Snowden affair, strongly held public concerns have the potential to damage the perceived legitimacy of vital bulk capabilities. Apart from a strong internal compliance and ethics culture, the best response to public concern is maximum possible transparency, consultation and strong ethical oversight.”

The report says the security services want to learn advanced techniques used to comb through masses of data for information about people, already used by companies such as Facebook.

“It is obvious however that developments in data-sharing and in discovery techniques (notably the increasingly sophisticated use of artificial intelligence and behavioural analytics to extract information from bulk datasets, alone and in combination) will require continuing legal and ethical review.

“The most sophisticated deployments of such techniques are not practised by intelligence agencies or police but by private sector operators including tech companies and major retailers. The world depicted in the film Minority Report remains strictly fictional. However, UKIC aspires to learn from the private sector, and if possible to catch up.”

Anderson continues: “Behavioural analytics is here to stay, and its techniques may be effective not just in refining the assessment of risk from existing leads and [existing suspects] but in discovering new leads who would not otherwise have come to the attention of the authorities.”

In his update, Anderson says MI5 and the police have made good progress in enacting recommendations meant to limit the chance of any errors and boosting their chances of catching terrorists before they strike.

The report also reveals that late last year MI5 formally took the lead from police in addressing the biggest threats posed by the far right. Anderson’s first report in December 2017 called for the same effort and tools to be used against the extreme rightwing threat as against Islamist terrorism.

Up until now it has been thought that Islamist terrorism generated far more plots and far more work for counter-terrorism investigators. But the fresh focus on the extreme right means investigators are latching on to potential threats earlier.

Anderson writes that there is “far greater interest” in weapons and military culture among extreme rightwingers than among Islamists. He says police “had been less likely than MI5 to see such interest as an indicator that the terrorism threshold had been reached”.

In his findings, Anderson says MI5 and the police are improving an already good working relationship. But a plan to share information with councils and local police about suspects, especially former suspects whose case has been closed, has run into problems, partly because of cuts to key local services.

Anderson said: “Some local authority representatives cautioned against unrealistic expectations of services such as mental health and community safety. It is not difficult to see how intensive interventions could assist in the management of closed SOIs [subjects of interest or suspects]; but against, what was described to me as, a background of widespread recent degradation of local services, such interventions may not be generally available, and there was a degree of reluctance in local authorities to prioritise closed SOIs at the expense of other citizens, or to take on the risk of any failure to do so.”