Far-right groups in the UK are at their lowest ebb for 20 years despite the apparently “favourable” factors of Islamist terrorism and the Rotherham abuse scandal, a watchdog has said, while also attributing the decline to internal splits and the rise of Ukip.

The annual report by Hope Not Hate, which has spent a decade monitoring far-right groups such as the British National party and English Defence League, stresses that Ukip is not itself of the far right. But the authors argue that the BNP has struggled after its leader, Nick Griffin, was ousted, and Ukip has “steamrollered through their previous heartlands and stolen their voters”.

The introduction to the report by Hope Not Hate’s chief executive, Nick Lowles, and Matthew Collins, a researcher, reads: “While Ukip is not the BNP and Farage is not Griffin, it is clear that most former BNP voters feel quite at home in the Ukip stable.”

However, the report warns that amid the general decline, a small but committed fringe of very young people are being drawn to extremist views, raising the risk of a “lone-wolf” attack. It also cautions that antisemitic incidents remain alarmingly common, fuelled last year in part by the Israeli military campaign in Gaza.

The report chronicles the fortunes of 24 groups, ranging from well-known names such as the BNP, EDL and English Democrats to tiny fringe organisations such as the South East Alliance, which is estimated to have 10 members, and the Nazi-sympathising Racial Volunteer Force, with an estimated strength of 25.

The publication was written before last week’s attacks in Paris which left 17 people dead. However, Lowles and Collins argue that the rise of Islamic State, and revelations about the abuse of around 1,400 girls by mainly Asian men in Rotherham, should have been a boon to far-right organisations.

They write: “And yet, despite these favourable circumstances, none was able to capitalise. They all had their demonstrations but they were relatively poorly attended and were overshadowed by violence as the various groups fought out their personal and political differences.”

Chief among the divided groups is the BNP, which recently boasted 58 councillors and two MEPs, figures now reduced to two and none. Among the departed MEPs is Griffin, since expelled from the party. The BNP, the report says, is “in a deep hole and unlikely to get out of it anytime soon”.

The EDL, which a few years ago organised large, high-profile protests against what it terms “Islamisation”, is now fragmented between regional factions, the report says, with total support of only 200 to 400 people. However, this could change if the group’s charismatic former leader, Stephen Lennon, also known as Tommy Robinson, returns to the helm.

Lennon stepped down in late 2013, saying the group had become too extreme. However, Lowles said, there were reports he could come back: “He is the one person currently around who could transform the fortunes of Britain’s far-right and anti-Islam groups. We are aware that many people are actively encouraging him to return and we feel there is a real possibility that he will.”

Another worry identified in the report is young activists drawn to fringe Nazi-sympathising groups like National Action. While tiny – it has no more than 40 main activists – the authors call NA “highly motivated, busy and politically coherent”. In October an NA sympathiser, 21-year-old Garron Helm, was jailed for sending anti-semitic tweets to the Labour MP Luciana Berger.

More generally, the report expresses alarm at antisemitism. In July last year, during the Gaza campaign, the Community Security Trust reported 302 antisemitic incidents, the most in a single month in the organisation’s 30-year history.

Lowles said: “Our report reveals that, despite otherwise favourable conditions, far-right hate networks are really struggling at the moment. For that we should all be thankful, though of course we must not rest on our laurels. As we have seen with events in France in recent days, the situation can change very fast.

“The rise of Ukip, as well as Nick Griffin’s own narcissistic downwards spiral, has certainly put paid to any hope the BNP had of electoral success. We must continue to scrutinise whether UKIP can truly distance itself from the ugly racism of Griffin’s former party.”