There’s a hard truth for anyone who stands on the left side of the political divide, a truth demonstrated again in recent days. Put starkly, the left usually loses. In the UK, Labour has governed for just 30 of the last 85 years. Of the eight men who have been president of the French fifth republic, only two have been socialists. In the postwar tally of US presidents, Republicans outnumber Democrats.

There are countless reasons for this, but here’s one to contemplate in a week when the reality of a fourth consecutive election defeat for Labour slowly sinks in (very slowly in some quarters), and as Democrats contemplate poll numbers showing that even as Donald Trump becomes only the third president in US history to be impeached he still enjoys an edge over his leading Democratic rivals in the battleground states likely to decide the election next November.

The tightly managed style of the Blair-Clinton era left voters weary of candidates who were polished and spun

A look at recent electoral history suggests an uncomfortable pattern. Not only is the left’s default setting to lose; it wins only when led by someone who is not merely good, but a phenomenon. In the US, it has taken a Bill Clinton or a Barack Obama – exceptional political communicators of once-in-a-generation talent – to break the Republican winning streak. In the UK, it took a Tony Blair – who all but the most stubborn critics will concede was an uncommonly gifted political operator – to become the only Labour leader born in the last 100 years to win a general election.

There is an asymmetry here, for the right can win power without offering a leader who dazzles. For parties promising to maintain the status quo and keep things ticking over, a George Bush (junior or senior) or John Major will do. That rarely cuts it for the centre-left. Unless the country is recoiling from a predecessor so discredited that voters long for an antidote whose very ordinariness becomes appealing – think Jimmy Carter after Watergate or François Hollande after Nicolas Sarkozy – the party of change usually loses unless it is led by a candidate of extraordinary charisma.

That’s also true of populist movements, who need a Trump, Matteo Salvini, Nigel Farage or Marine Le Pen figure at the top because, unlike regular parties of the right, “it’s all they’ve got. They can’t promise governing competence or stability,” says the Labour peer Andrew Adonis, who has studied leadership closely. But the burden falls especially heavily on the left. They are demanding the nation give up what’s familiar and head instead into uncharted territory. That requires a charismatic leader. For, to cite (again) the words of an old teacher of mine, the late Ze’ev Mankowitz, “People don’t believe in ideas: they believe in people who believe in ideas.”

And yet, for a quality so determinative of political success or failure, it is maddeningly hard to define charisma, even if we feel we can spot its presence – or absence – in seconds. At its most basic, it begins with elementary communication skills. Would a TV channel choose this or that candidate to read the news? Applying that criterion to 1994, you’d have known in an instant that, had it come to a contest, Blair would have beaten Gordon Brown. Obvious.

But there is so much more to it than that. Charisma is a shifting, evolving quality, changing in reaction to the times and especially to whatever has gone before. The tightly managed political style of the Blair-Clinton era left voters weary of candidates who were polished and spun, hankering instead for something apparently more authentic, even dangerous. Today the key TV test is whether you’d turn the sound up to hear what they say, not least because you wouldn’t be sure what was about to come out of their mouth.

Trump is the master of that form of unpredictability; Boris Johnson built a career on the frisson of appearing to defy convention and say things politicians aren’t meant to say. Of the current Labour field, the only candidate who even comes close to having that quality is Jess Phillips. Stewart Wood, the former adviser to Gordon Brown and Ed Miliband, neither of whom were famed for their charisma, can see Phillips appealing partly because voters will know that “she’ll sometimes screw up and say the wrong thing. She’s visceral. She’s got that unstable, combustible mix”, which can be compelling.

For a candidate of the right, that alone can carry you to victory. But it’s so much harder for the left. They have both to excite and reassure, all at the same time. Clinton, Blair and Obama succeeded partly because they calmed those who would normally be resistant to change. To take a niche, but illustrative example: in Israel, the centre-left has won only twice in the last 42 years, on both occasions when they were led by certified military heroes. Both Yitzhak Rabin and Ehud Barak had the credentials to soothe any doubts about security. They were phenomena. Anything less, and their party loses.

In less extreme form, it means a left challenger needs to exude gravitas and competence. The model here is Clement Attlee, who could reassure Britons in 1945 not least because he had been at Winston Churchill’s right hand as deputy prime minister throughout the war. What he lacked in sparkle, he made up in heft. His was the kind of anti-charisma offer Brown’s team were aiming to make with their “Not flash, just Gordon” slogan in 2007. We will never know if it would have worked in the election that never was, after which Brown’s image was no longer his to shape. But a left leader has to have at least one of those qualities – authority or charisma – and ideally both. (Supporters of Keir Starmer reckon his record as director of public prosecutions gives him a weight his rivals lack.)

What complicates this further is that charisma can be in the eye of the beholder. To me, Jeremy Corbyn always looked like a glowering supply teacher, a man incapable of uttering a single memorable sentence, unable to command a room. In the campaign just gone, it became clear that millions of voters saw him that way too. Yet to thousands of Labour activists, he was an inspiration. What might account for this perception gap is a notion of charisma confined mostly to the left.

Corbyn clearly met the post-Blair demand for an unspun, authentic politician, but he also cut a figure with deeper historical resonance. Ascetic, living modestly, with a scraggly beard trimmed only once he won the leadership in 2015, it was easy to see him as the archetypal prophet, summoned from the wilderness after decades spent living in isolation with only his principles for company. Never mind that he’s a boring or repetitive speaker: that only proves his consistency. Never mind the venom of his detractors or even the serial defeats: that only confirms him as a holy man who refuses to bend.

Bernie Sanders has some of this appeal in the US, but here’s the key point: for all its roots in religious tradition, this is a form of charisma that today stirs only the left. Study the photograph on the front of Friday’s Guardian. It shows a gaunt Corbyn, staring fixedly ahead, alongside a chunky, Falstaffian Johnson, a mischievous smile on his lips. It’s an image as old as England, Roundhead against Cavalier - and we know which side the country reverts to in the end.

Democrats need to hold this structural imbalance between left and right in mind as they choose their challenger to Trump. Unfair it might be, but to win that person will need to offer both excitement and solidity, hope and calm. It’s not an easy combination to pull off. Labour is in need of a similarly rare blend. But first it needs to decide what question it is asking. Is it looking for a leader who ticks all the members’ pet ideological boxes, or for someone who might prove capable of interrupting Labour’s long losing streak? By their choice, we’ll know whether they want to win – or to enjoy once more the martyr’s thrill of glorious defeat.

• Jonathan Freedland is a Guardian columnist