If this Tory parody of a government is so shambolic, so chronically divided, so utterly directionless, then why no astronomical Labour lead in the polls? If we have a prime minister in name only, afflicted by cabinet resignations, presiding over falling wages and a chaotic Brexit process, then why has Labour not opened up a 20-point advantage over its crisis-stricken rivals?

It must be said that, in the different political era that was April, Labour polling more than 40% and holding a consistent but small lead over the Tories would be considered a nice problem to have. The party entered the general election campaign on 24% – half the Tories’ poll rating. Labour, it was widely predicted, would slump to its smallest parliamentary caucus since 1935. Instead, it added 10 percentage points to its share in 2015, a surge without precedent since Clement Attlee more than seven decades earlier. What should disrupt the sleep of Tory MPs now is what happens if Labour goes into the next general election starting on more than 40%.

Such is the poverty of Tory ideas that they deny poverty even exists | Zoe Williams Read more

Set against the crisis of European social democracy, British Labour represents an electoral near-miracle. Labour’s sister parties have leaderships that are far closer in political outlook to the old third way dogma than they are to Corbynism. They are in a catastrophic position. Germany’s Social Democrats amassed a pitiful 20% in the country’s recent elections; the Austrian Social Democrats achieved less than 27% in October’s elections; Spain’s Socialists languish in the mid-20s. Still, it could be worse: a humiliating 6% for the French Socialists in the first round of the presidential elections, and a near-terminal 5.7% for the Dutch Labour party.

One striking exception is Portugal’s ruling Socialists who – under pressure from radical leftwing parties – have reversed some austerity measures. Labour’s relative success compared with its floundering sister parties is of little comfort for those languishing under a Tory government, though. So why no colossal lead?

The truth is, Britain has polarised into two hostile camps, divided by generation, security and world-views. That is why the two-party system has reasserted itself, with the combined Tory-Labour share of the vote at its highest level since 1970. In the past 43 years, Labour has only twice won a higher share of the vote than it managed in June. One of those, New Labour’s 2001 landslide, was a mere 0.7 points higher. But the Tories, too, won their best share of the vote since 1983. Labour boasts a Blair-landslide poll share, but the Liberal Democrats have almost vanished and the Tory vote has solidified. And so: stalemate.

Here’s why. The Tories won a 50-point lead among those aged over 70, and a 31-point lead among Britons in their 60s. Social democracy has, ironically enough, been largely preserved for pensioners. That is not a battle cry for a generational war; the quarrel with the Tories should not be over preserving the security of older Britons, but for attacking the security of everybody else. Pensioners’ household income in 2016 was three times higher than 40 years earlier, compared with twice as high for working-age Britons. According to the Resolution Foundation, the typical pensioner has a higher income after housing costs than those in work.

Play Video 4:41 Watch Emily Thornberry rattle Damian Green at PMQs – video highlights

In the years following the crash – and the implementation of austerity – pensioners’ incomes actually rose by 10%. Take housing, too. Home ownership among the over-65s jumped from 63% in 1997 to 77% last year.

Let’s not forget the 1.9 million pensioners languishing in poverty. But there has not been an onslaught against the conditions of pensioners that might fuel a desire for a decisive break with the status quo.

Labour, on the other hand, won 61.5% of the vote among the under-40s. And little wonder. Working-age Britain has endured the worst slump in wages of the industrialised world other than Greece. Home ownership for younger Britain is in a state of collapse: from 54% two decades ago to just 34% last year. Optimism about the future has sapped away for younger Britons: just a third of under-35s believe they will have a better life than their own parents. Lacking security and hope, a radical alternative naturally appeals.

To win even a sliver of extra support among pensioners would make a decisive difference

And then there are values. The Tory coalition is bound by relative security and social conservatism. Just 13% of those over 65 believe it would be acceptable for a royal to marry someone of the same sex, compared with two-thirds of those aged between 18 and 24. While fewer than half of adults younger than 25 want immigration tightened, the figure rises to an overwhelming 84% among those over 65.

A poll in 2015 suggested that while just 17% of those aged 18 to 24 thought immigrants should “leave behind their own cultures and traditions”, the figure was nearly half among those older than 60. More than four in 10 younger Britons think Islam is compatible with British values; just over a fifth of older Britons agree.

There are two Britains living in almost parallel universes. One feels that the social order is broken and needs replacing; the other does not, and indeed fears the alternative. One is relatively comfortable with immigration, Muslims, LGBT rights; the other is deeply uneasy. And that is why there is such a profound polarisation which – structurally – makes it so hard for Labour to achieve a commanding lead. The party has two real options. First, while turnout markedly increased among younger Britons, it remains lower than it was in the 1980s. Labour’s support among working-class younger Britons is overwhelming – but it is here turnout is lowest. A relentless voter registration drive – as Labour launched after the election was called – could hardly be more critical. Second, to win even a sliver of extra support among pensioners would make a decisive difference. A relentless focus on pensioners’ issues – from social care to pensions to loneliness and solitude – is of paramount importance if Jeremy Corbyn is to assume the premiership.

Unless Labour gets out the working-class youth and wins over pensioners, the next election isn’t in the bag. A small win, too, will pose problems: first, because it will only take a handful of ideologically opposed Labour MPs to block radical legislation; and two, governing parties tend to lose seats, and one term of a radical Labour government is insufficient to transform society. The plight of centrist-led social democratic parties across the Channel underlines that Labour’s only hope is a radical approach.

• Owen Jones is a Guardian columnist