Political developments in Britain appear more than a little confusing at the moment.

The parliamentary Labour party is in open revolt against a leader recently elected with the biggest mandate in the party’s history. Most Labour MPs call Jeremy Corbyn “unelectable”, even though they have worked tirelessly to undermine him from the moment he became leader, never giving him a chance to prove whether he could win over the wider British public.

Now they are staging a leadership challenge and trying to rig the election by denying tens of thousands of Labour members who recently joined the party the chance to vote. If the MPs fail in the coming election, as seems almost certain, indications are that they will continue their war of attrition against Corbyn, impervious to whether their actions destroy the party they claim to love.

Meanwhile, the Guardian, the house paper of the British left – long the preferred choice of teachers, social workers and Labour activists – has been savaging Corbyn too, all while it haemorrhages readers and sales revenue. Online, the Guardian’s reports and commentaries about the Labour leader – usually little more than character assassination or the reheating of gossip and innuendo – are ridiculed below the line by its own readers. And yet it ploughs on regardless.

The Labour party ignores its members’ views, just as the Guardian ignores its readers’ views. What is going on?

Strangely, a way to understand these developments may have been provided by a scientific philosopher named Thomas Kuhn. Back in the 1960s he wrote an influential book called the Structure of Scientific Revolutions. His argument was that scientific thought did not evolve in a linear fashion, as scientific knowledge increased. Rather, modern human history had been marked by a series of forceful disruptions in scientific thought that he termed “paradigm shifts”. One minute a paradigm like Newtonian mechanics dominated, the next an entirely different model, like quantum mechanics, took its place – seemingly arriving as if out of nowhere.

Importantly, a shift, or revolution, was not related to the moment when the previous scientific theory was discredited by the mounting evidence against it. There was a lag, usually a long delay, between the evidence showing the new theory was a better “fit” and the old theory being discarded.

The reason, Kuhn concluded, was because of an emotional and intellectual inertia in the scientific community. Too many people – academics, research institutions, funding bodies, pundits – were invested in the established theory. As students, it was what they had grown up “knowing”. Leading professors in the field had made their reputations advancing and “proving” the theory. Vast sums had been expended in trying to confirm the theory. University departments were set up on the basis that the theory was correct. Too many people had too much to lose to admit they were wrong.

A paradigm shift typically ocurred, Kuhn argued, when a new generation of scholars and researchers exposed to the rival theory felt sufficiently frustrated by this inertia and had reached sufficiently senior posts that they could launch an assault on the old theory. At that point, the proponents of the traditional theory faced a crisis. The scientific establishment would resist, often aggressively, but at some point the fortifications protecting the old theory would crumble and collapse. Then suddenly almost everyone would switch to the new theory, treating the old theory as if it were some relic of the dark ages.

Science and politics are, of course, not precisely analogous. Nonetheless, I would suggest this is a useful way of understanding what we see happening to the British left at the moment. A younger generation no longer accepts the assumptions of neoliberalism that have guided and enriched an elite for nearly four decades.

Ideas of endless economic growth, inexhaustible oil, and an infinitely adaptable planet no longer make sense to a generation looking to its future rather than glorying in its past. They see an elite with two heads, creating an illusion of choice but enforcing strict conformity. On the fundamentals of economic and foreign policy, the Red Tories are little different from the Blue Tories.

Or at least that was the case until Corbyn came along.

Corbyn and his supporters threaten a paradigm shift. The old elites, whether in the Labour parliamentary party or the Guardian editorial offices, sense the danger, even if they lack the necessary awareness to appreciate Corbyn’s significance. They will fight tooth and nail to protect what they have. They will do so even if their efforts create so much anger and resentment they risk unleashing darker political forces.

Corbyn’s style of socialism draws on enduring traditions and values – of compassion, community and solidarity – that the young have never really known except in history books. Those values seem very appealing to a generation trapped in the dying days of a deeply atomised, materialist, hyper-competitive world. They want change and Corbyn offers them a path to it.

But whatever his critics claim, Corbyn isn’t just a relic of past politics. Despite his age, he is also a very modern figure. He exudes a Zen-like calm, a self-awareness and a self-effacement that inspires those who have been raised in a world of 24-hour narcissism.

In these increasingly desperate times, Cobyn’s message is reaching well beyond the young, of course. A paradigm shift doesn’t occur just because the young replace the old. It involves the old coming to accept – however reluctantly – that the young may have found an answer to a question they had forgotten needed answering. Many in the older generation know about solidarity and community. They may have been dazzled by promises of an aspirational lifestyle and the baubles of rampant consumption, but it is slowly dawning on them too that this model has a rapidly approaching sell-by date.

Those most wedded to the neoliberal model – the political, economic and media elites – will be the last to be weaned off a system that has so richly rewarded them. They would rather bring the whole house crashing down than give Corbyn and his supporters the chance to repair it.