There are several mysteries in British politics at the moment. Here is one that Sherlock Holmes would call a three-pipe problem: why are the Lib Dems doing so atrociously?

On the face of it, the weather is set fair for a classic third-party revival. The Tories and Labour are both fighting civil wars. The Conservatives are riven by vicious infighting over Brexit; Labour’s angry summer has been devoured by a shameful battle about antisemitism. The two leaders of the major parties are not liked by most of their own MPs. Many voters who self-identify as Labour or Tory say they are dissatisfied with those parties. When asked to choose between Theresa May and Jeremy Corbyn as the best candidate for Number 10, the most popular choice is Neither. More than half of voters say that no party represents their views. A very substantial chunk of the electorate has no affinity with the ugly fanaticism of either Corbynite zealots or the Brextremists. That ought to leave many empty hectares of prime political real estate available to be snapped up and built on.

Yet Britain’s traditional party of the middle ground is flatlining. The Liberal Democrats are polling in poor single digits, perilously close to the point where their rating is indistinguishable from the margin of error. They put on more council seats than any other party in the local elections in May and have attracted more members since Brexit, but it is the national ratings that get the attention. Rightly so, when the Lib Dems still claim to have aspirations to be back in government one day. These ratings are dreadful in themselves and even worse in the context of simultaneous crises afflicting both the larger parties.

This miserable showing has generated a growing rumble about the leadership of Sir Vince Cable. This threatened to be the main narrative when the party conference kicks off in Brighton next weekend. In so much as the media was going to pay that gathering any attention at all, a lot of journalists would be in attendance only to spend their time trawling the seafront for Lib Dem activists ready to tell their ballroom-dancing leader that he ought to foxtrot off.

Sir Vince was trying to preempt that becoming the story of his conference when, on Friday, he told them not to bother discussing his retirement because he had already decided on it: he will be quitting before the next election unless one should happen in the near future. Though he did not offer a precise timeline for his departure, he said he would stand down once Brexit was “stopped or resolved”. This gives him some wriggle-room to stretch out his swansong. “Once Brexit is resolved” is a rather elastic formula. But it doesn’t sound compatible with leading the Lib Dems any further than 2020 and most of his colleagues expect him to be gone some time in 2019.

This is a recognition that the task of reviving the Lib Dems has proved to be more demanding than he anticipated. This time last year, when political editor Toby Helm and I interviewed him on the eve of his conference, Sir Vince was suffused with so much optimism (or delusion) that he said he did not think it impossible that he could become prime minister. He’s since had a year to realise that this isn’t going to happen.

With Sir Vince’s future (more or less) settled, that leaves us with a much larger question of interest to anyone despairing of the current party alignments. Why have the Lib Dems singularly failed to capitalise on what look like highly propitious circumstances for them?

One possible explanation is that they are still toxified by their five years in coalition with the Conservatives. This is an obvious explainer, but now an outdated one. The coalition seems very long ago. Pre-Brexit, pre-Corbyn, pre-Trump, the coalition government of David Thingy has the cobwebbed feel of another epoch. Any lingering memories of those years are surely being effaced by the dysfunctionality of the current government and its Labour opposition.

A better explanation for why the Lib Dems are struggling so badly is that they simply lack heft. The third party always has to fight to get a word in edgeways. That handicap is compounded when the voice has become enfeebled. Two bad general elections have left the Lib Dems with just a dozen MPs. Of them, Sir Vince himself is the only one with much national name recognition. There comes a point when a party falls so low in its parliamentary representation, which is then reflected in scant media attention and voter interest, that it just doesn’t get taken very seriously. That is the view of more than one former Lib Dem leader to whom I’ve spoken.

Then there is the starkness of the divide between the two main parties as Corbynism tightens its grip on Labour and Rees-Moggery becomes rampant in the Tory party. It may be wrong to think that this polarisation ought to make it easier for the Lib Dems to revive as a voice of moderation. It may actually be making it harder to do this. Our electoral system imposes a forced choice on many voters. They feel compelled to pick between the horribly unappetising and the absolutely unacceptable. Imagine you are a voter who is utterly repelled by the Tories, but who also thinks that a Corbyn premiership would be a calamity for Britain. If your top priority is stopping the Labour leader from getting to Number 10, you would probably put aside the Lib Dems as an option on the grounds that the only reliable way to prevent a Corbyn premiership is to hold your nose and back the Tories. It works the other way around. Imagine you are a voter who loathes Labour’s current leadership, but who also thinks the Tories are a disaster for Britain. Again, even if you rather warm to the Lib Dems and really don’t fancy Labour, you might well end up casting for Labour as the best way to block the Tories. This would explain why the combined vote share of the top two parties was so high at the last election, even when so many voters expressed dissatisfaction with both of them.

The only answer to this conundrum is for a third force to look like a serious alternative to making the devil’s choice. The Lib Dems seem to know that they are too weak to achieve this on their own. Witness how they talk about the prospect of a new party emerging from a breakaway of Labour moderates, possibly combined with a breakaway from the Tories by some Conservative MPs. The Lib Dems have not poured scorn on the notion of a fresh, broadly centrist formation, even though it would be a competitor in their political space. Sir Vince and other senior Lib Dems have actively encouraged the idea. He says openly that he has been talking to disenchanted Labour and Tory MPs about the possibility of something new emerging.

His other idea is to reach out to moderate voters who are not currently engaged by his party or any other. He wants to create something he calls a “movement for moderates”. This is easy to ridicule. What do we want? The centre ground! When do we want it? When the time seems appropriate! How do we want it? In carefully considered increments!

It is easy to mock, but history shows that it can be done. With an appealing leader armed with attractive ideas and operating in a promising context, it is perfectly possible to inspire passion and mobilise enthusiastic support from a broadly centrist position. Tony Blair did that in Britain when he won by a landslide in 1997. Barack Obama did it in America in 2008. Emmanuel Macron did it, with a lot of help from his opponents and the French presidential voting system. Justin Trudeau followed the same successful formula when he led the Canadian Liberals out of the wilderness and into power three years ago.

Some kind of new formation in British politics is entirely possible. It will need a leader with a rare combination of gifts: sufficient experience to be taken seriously, enough freshness to be exciting, and a body of smart ideas that are both galvanising and credible. The examples of Blair, Obama, Macron and Trudeau suggest that a bit of charisma, a sprinkling of stardust, would be a great help in animating support. A talent for inspiring confidence will also be crucial.

Sir Vince is not wrong to think that, with the right leadership, there is an untapped appetite for a revival of the centre ground. He is also not wrong to think that, if the person exists to make this happen, it is not him.

• Andrew Rawnsley is an Observer columnist