It is a grim reflection that no contemporary British politician better understands the networks, dynamics and ever-changing rules of modern politics than Nigel Farage.

He is as effective as he is awful. His fledgling Brexit party is not fielding candidates in Thursday’s local elections – though its baleful spirit will surely loom over the battle for control of 248 English councils, in which the Tories are expected to suffer serious losses.

Instead, Farage is keeping his powder dry for the European elections that will be held on 23 May, assuming Theresa May has not secured Britain’s exit from the European Union by then.

In the latest Opinium poll of voting intention, his new movement is level pegging with Labour on 28% – 14 points ahead of the Tories.

This is merely embarrassing for Labour, as the party that hopes to form the next government. For the Conservatives, a true catastrophe looms. I am struck by the number of senior Tories who are simply stultified by the disaster engulfing them. “What should I do?” one experienced and normally stoic MP said to me last week. “I just have no idea what to do.”

The myth of the old soldier returning from private life for England is a good one – and he mobilises it well

Farage is indeed the nemesis of the party he tried repeatedly to represent at Westminster but finally left in 1992, disgusted by the Maastricht treaty. It is not quite true that he single-handedly drove David Cameron to embrace a referendum on the UK’s membership of the EU. But there is no doubt that Ukip’s surge under Farage’s leadership – and its consequences for Conservative unity – was uppermost in Cameron’s mind when he initiated the six-year saga that became Brexit.

As if to confirm his worst fears, Ukip went on to win the highest share of the vote in the 2014 European elections. Farage, since becoming leader in 2006, had transformed his party from a marginal club of cranks obsessed by national sovereignty and the minutiae of EU directives into a cultural movement united in its fixation with, and opposition to, immigration.

In so doing, he wrote the horrible script for the 2016 referendum, in which his Leave.EU campaign acted as the provisional wing of the pro-leave cause.

While Boris Johnson and Michael Gove promised bounteous trade deals, an NHS spending bonanza and a fresh start for the liberated UK, Farage poured poison into political discourse – most notably with his vile poster showing a long queue of Syrian refugees under the slogan “breaking point”. It was seriously nasty, and it worked.

Facebook Twitter Pinterest ‘He wrote the horrible script for the 2016 referendum.’ Brexit party rally at Clacton-on-Sea, Essex. Photograph: Martin Dalton/Rex/Shutterstock

Who is this man? A talkshow host, a spiv, a political groupie, a hanger-on – as likely to visit Julian Assange as Donald Trump. Watch his toe-curling performance talking to Steve Bannon in the recent documentary The Brink for a study in fidgeting awkwardness (Farage knows this is not a good look) mixed with irrepressible adulation (he hangs on Bannon’s every word).

His methods are contemptible – but successful. He uses his membership of the parliament of a supranational organisation he wants the UK to leave, to give him funding and access to public service broadcasting. His fawning relationship with Trump has bequeathed him (unbelievably) something close to global reach.

There are many facets to Farage’s success – he and many others of the populist right have been conspicuous beneficiaries of the 2008 financial crash. But his talent, I believe, lies in a deep, mostly instinctive grasp of political narrative and its operations in the digital era. Farage knows that simple stories, driven home relentlessly, can be spectacularly successful if they answer a collective yearning. Hence, his claim to have to “come out of semi-retirement”, answering a great historic calling, with a measure of reluctance but unfailing patriotism.

This is, of course, ridiculous. Far from being “semi-retired”, Farage has been almost impossible to avoid since he stepped down as Ukip leader: a constant presence on the airwaves, in print and online. But the myth of the old soldier, returning from private life to perform one last service for England, is a good one – and he mobilises it well.

No less resonant (and pernicious) is the great Brexit betrayal myth that Farage, more than any other politician, has cultivated – and since before the referendum itself. As Mark Lilla argues in his book, The Shipwrecked Mind: On Political Reaction: “Where others see the river of time flowing as it always has, the reactionary sees the debris of paradise drifting past his eyes.” Such a figure is animated by “the militancy of his nostalgia”, which, Lilla suggests, makes him “distinctly modern”.

Farage raises hackles in Oldham by saying town is split on racial lines Read more

In this respect, it was always baked into Farage’s rhetoric that if the public voted to leave, the elite would seek to thwart their wishes. The truth, again, is quite otherwise. The political class has strained every tendon to find a way of delivering the undeliverable: of extracting the UK from a 46-year relationship without wrecking its prosperity, security and access to the wider world.

Brexit has failed because the square-circling task is impossible. We must stay, or accept a grievous cost: that is the choice now. But Farage appeals to a primal social instinct: the sensation that the few are, yet again, cheating the many of their unsullied dream. It is not the dream that is at fault, you understand, but those who sabotage it. Just as Marxists insist true communism has never been tried, so Brexiteers declare that their simple plan has been wrecked by weaklings, quislings and fools.

Brexit was designed by its most passionate supporters to fail: its purpose was to be betrayed, to enable a new movement to rise up, animated by fury and fear. Such a movement has now been born. It is already tearing the Conservative party to pieces. That, sad to say, is only the beginning of its plan.

• Matthew d’Ancona is a Guardian columnist