The UK’s police and security services remain dangerously ill-equipped to identify and counter the long-term terrorism threat from a new wave of far-right politics, according to researchers who helped head off a plot to assassinate a Labour MP.

The anti-racism charity Hope not Hate was instrumental in foiling a plan to murder the West Lancashire MP Rosie Cooper. The legal case at the centre of this plot came to a conclusion on Tuesday.

Senior security figures including the head of MI5, Andrew Parker; the head of UK counter-terrorism policing, Neil Basu; and the Metropolitan police commissioner, Cressida Dick, have warned that the far-right threat is growing rapidly.

But Hope Not Hate (HNH), whose mole Robbie Mullen exposed the plot to kill Cooper after he turned against the banned neo-Nazi group National Action (NA), said the security services and police were failing to keep up with potential terrorists.

“They don’t know where the threat is coming from and don’t know where to find it, and then they lump in what they think is a threat from the far left,” said Matthew Collins, head of intelligence at HNH, who was also Mullen’s handler.

Jack Renshaw, the white supremacist who planned to kill the Labour MP Rosie Cooper. Photograph: BBC

After the effective destruction of NA – which was the first far-right group to be banned in Britain since the second world war – HNH warned of the long-term threat from a young generation of violent neo-Nazis who have emerged from splits in the far right.

The charity cited groups such as the Sonnenkrieg (Sun war) Division, which HNH described as the third generation of NA.

“The far-right terror threat has always been there. People in groups like C18 have always had sick dreams and fantasies of killing black and Asian people, but they rarely went and did it because they didn’t want to die themselves or go to prison for a long time,” said Collins.

“That’s why National Action and the Atomwaffen [Nuclear weapons] Division, an American group, have been developing the idea of a ‘white jihad’. That was to persuade and convince people that there was spiritual reason for why they could die or go to prison.”

The police have been slow to grasp such concepts, he added, and have largely still been reliant on tipoffs, 999 calls out of the blue, or terrorists making mistakes.

“Those who are involved in groups like the Sonnenkrieg Division have been meeting on gaming forums and other places. The groups that are picking up where NA left off are smaller and more dangerous. If one gets banned they then pick up the mantle and run with it.”

According to academics, extremist groups are using the resurgence of rightwing politics, particularly online but also on the streets, as friendly territory within which they can act more openly and recruit.

“Globally we are seeing the rise of right populism,” said Jacob Davey, a researcher at ISD Global, which helps to design and implement counter-radicalisation strategies. “We can see more extreme groups pushing and pulling and engaging with these and using this as an opportunity.”

Individuals and groups with hard-right ideologies infiltrate into groups that they see have parallel, but legitimate grievances, he said. “The Football Lads Alliance initially had thousands of people at their demonstrations and I would have hesitated to call that a far-right movement; I think it was a lot of people who were legitimately concerned about terror. But what you see is entry into these groups by the far right.”

Online communities developing around these radical, but comparatively soft rightwing ideologies are also used by more extreme groups as a gateway to direct potential recruits harder material hosted on private chats hosted on WhatsApp, Telegram and Discord servers.

“Social media is crucial to the current expansion of the extreme right,” Davey said. “It’s lowered the barrier to getting involved in the extreme right. Twenty years ago if you were that way inclined … you had to go to a certain pub, you had to go to make your way to a certain rally.

“But also you [were] more publicly aligned with these groups, you might lose your job, you might lose friends or family.”

A shared culture links adherents, rather than loyalty to one particular group, said Paul Jackson, a historian who specialises in the history of fascism. The most extreme coalesce around an ideology – revolutionary ultranationalism or racially driven politics – that rejects liberal democracy, and distinguishes them from rightwing figures such as Nigel Farage or Tommy Robinson.

“This type of neo-Nazism is more intellectual than perhaps you may find within neo-Nazis of the Blood and Honour white power music scene, which used to be the youth culture, but now they are getting on to their 50s and 60s,” Jackson said.

“It likes to see itself as extreme within the wider far-right milieu. National Action would comment disparagingly on the English Defence League and the BNP as dad-like figures who didn’t have the necessary dynamism.”

However, they were more of a “talking shop revelling in the idea of violence” rather than an actual violent extremist cell planning attacks, Jackson said. “The danger is that people on the fringes decide to act by themselves. That’s what we find with people like [Jo Cox’s killer] Thomas Mair or Anders Breivik.”

The number of people referred to the UK government’s counter-extremism programme over concerns about far-right activity has risen by more than a third, recent figures show.

A Home Office spokesperson said: “Our counter-terrorism strategy, Contest, addresses all forms of terrorism and no individual or group is free to spread hate or incite violence. We do not routinely comment on whether organisations are or are not under consideration for proscription.”