Ministers have vowed to overhaul Britain’s fight against terrorism after a report revealed chances to thwart the Manchester attack were missed and the leader of the London Bridge assault struck while under investigation by MI5 as a threat to national security.

The findings followed this year’s spate of atrocities that killed 36 people, and come as the level of threat is assessed by counter-terrorism experts to have markedly increased, with warnings more attacks will get through Britain’s defences.

Internal reviews by MI5 and the police cleared themselves of making serious mistakes that allowed terrorists to strike. But a summary of the findings revealed that the attack on Manchester Arena that killed 22 people might have been prevented if different decisions had been made by MI5.

The security agency had intelligence about the suicide bomber, Salman Abedi, whose “true significance was not appreciated at the time”, a report by the barrister David Anderson QC said. He said that with hindsight, different decisions might have been taken, but it was unknowable whether Abedi would then have been stopped. He declined to endorse MI5’s view it would not.

Reviews of the four UK terrorist attacks between March and June this year were overseen by Anderson to provide assurance to the government that the internal reviews by the police and MI5 were thorough enough.

The attacks came as MI5 faced a surge in jihadist activity, facing more plots in the first half of 2017 than in all of 2016.

The plots thwarted since March now have reached nine, with two more charged on Tuesday with an alleged attack plot.

Salman Abedi’s significance was ‘not appreciated at the time’. Photograph: AP

The attack that MI5, which leads intelligence gathering on counter-terrorism, is most troubled by was the one in Manchester, where Abedi detonated a suicide bomb in the worst atrocity in Britain since the 2005 attacks on London’s transport system.

Anderson said: “It is conceivable that the Manchester attack in particular might have been averted had the cards fallen differently.”

In the weeks before the Manchester attack on 22 May, an exercise to examine which of 20,000 former terror suspects were worth further inquiry identified Abedi as “one of a small number of individuals, out of a total of more than 20,000 closed SOIs [subjects of interest], who merited further examination”. But the meeting to discuss this was not scheduled until 31 May.

Intelligence received by the counter-terrorism unit for the north west was not fully appreciated and thought to concern gang-related criminality, not terrorism.

Anderson said: “On two separate occasions in the months prior to the attack, intelligence was received by MI5 whose significance was not fully appreciated at the time. It was assessed at the time not to be [related to] terrorism but to possible non-nefarious activity or to criminality on the part of Salman Abedi.

“In retrospect, the intelligence can be seen to have been highly relevant to the planned attack.”

But the volume of work faced by counter-terrorism investigators means resources are being shifted as suspects rise and fall in the danger they are assessed to pose. “The reason why the judgements can be tough is that they are made against a background of imperfect information, and yet frequently require staff to choose which of a number of current and potentially deadly threats is more deserving of scarce investigative resources,”, Anderson said.

From Anderson, the home secretary and a family of one of those murdered, there was sympathy for the complexity of the task faced by MI5 and the police.

Dan Hett, whose brother Martyn died in the Manchester attack, said: “I can’t imagine the kind of complexity that these services deal with, and the level of decision that they need to make constantly ... Applying 20/20 hindsight to a difficult and stressful scenario is tempting but needs to be framed within the wider context of what they’re dealing with daily.”

Khuram Butt was under active MI5 investigation when he struck with the two other London Bridge attackers in June, and was suspected of potential involvement in attack planning. He was also on bail for criminal matter.

He was an acolyte of the extremist preacher Anjem Choudary, himself jailed for urging support of Islamic State.

Butt was described as increasing his operational security and by 2017 worked at the Ummah fitness centre in east London, where he met his fellow attacker Rachid Redouane. He taught Qur’an classes to youngsters alongside his other co-conspirator, Youssef Zaghba.

MI5, in the weeks before Butt and the two others struck at London Bridge, was still trying to gauge what danger he posed.

Anderson said: “The unpalatable lesson of London Bridge is that even priority subjects of interest in respect of whom sound decisions are being made ... may retain the ability to conceal their attack planning from the authorities.”

The reviews concluded that Khalid Masood, the Westminster attacker, and Darren Osborne, the Finsbury Park attacker – who was motivated by an extreme rightwing ideology, could not have been stopped, even with hindsight. Masood was the third person known to MI5 to attack in 2017 and was a former suspect, who used YouTube to help prepare his attack.

Anderson stressed the warning from security and police chiefs that not everything can be stopped. The attackers behind Westminster and Finsbury Park “would have been far from the top of any rationally compiled threat grid are always likely to emerge and strike opportunistically with readily available weapons”, with the deaths inflicted by car, knives and readily available chemicals to make Abedi’s bombs.

Candles lie along Westminster Bridge following the terror attack in March. Photograph: Andy Rain/EPA

In the Commons, home secretary Amber Rudd Rudd signalled a shift in the government’s approach, promising counter-terrorism policing would have the funding it needed and confirming her intention to bring in new counter-terrorism laws next year.

The new emphasis includes much greater sharing of intelligence data between the security services and local authorities, with the first multi-agency panel to be piloted in Manchester. She said there would also be a new approach to managing domestic extremism, particularly extreme rightwing groups, where their activity met the definition of terrorism. Greater attention will be made to assess data available on suspects past and present, such as their online activity.

Rudd kept open the possibility of a further inquiry and parliament’s intelligence and security committee is expected to investigate.

Counter-terrorism policing is facing a 7% budget cut and police chiefs warn neighbourhood policing, crucial to getting intelligence, is threatened by a financial squeeze.

Manchester mayor Andy Burnham said: “Real-term cuts to the police budget must stop and increases are given to reflect the greater work load and the growing threat,” he said.

Anderson said the reforms were more urgent given the increased terrorist activity: “In an increasingly high volume business, where success and failure depend on tiny margins, there will almost certainly be future cases in which these recommendations tip the balance in favour of the security forces.”