As the Jeremy Corbyn project hurtles towards an existential election, an old alliance is reasserting itself. The political partnership between Corbyn and John McDonnell sustained the Labour left at its weakest moments. “Jeremy is my best friend in parliament,” McDonnell used to say, before being swiftly corrected by his wife, Cynthia Pinto: “Your only friend!”

This explains why, in part, Corbyn was chosen as the Labour left’s flagbearer in the 2015 leadership contest: while the affable backbencher was then almost universally liked by his colleagues – one or two things have happened since – McDonnell was contemptuous towards Labour MPs he believed were spineless and unprincipled. That contempt was reciprocated in kind: when McDonnell attempted to stand for leader in 2007, Tom Watson wrote a blog wondering why the left was fielding someone so discourteous to his colleagues. Surely it made more sense, he suggested, for them to field the kindly Corbyn instead.

In the Labour left’s long wilderness years, the Corbyn-McDonnell axis functioned like this. McDonnell was the working-class intellectual, found in parliamentary corridors clutching a well-thumbed copy of Parliamentary Socialism, a tract written by Marxist intellectual Ralph Miliband excoriating the party’s lack of radicalism. At the heart of his operation was Andrew Fisher, who became Labour’s head of policy and was the author of the party’s 2017 manifesto: together, they convened politically sympathetic economists and wrote alternative budgets in an attempt to solve the lack of intellectual vision on a Labour left seemingly in terminal decline.

Corbyn was more of a “socialism from the gut” politician: his beliefs had more to do with an emotional revulsion at injustices such as homelessness and poverty than socialist literature. His interest in international issues – from Palestine to Sri Lanka – led him to be regarded as a sort of “foreign secretary of the left” and the campaigning zeal he later displayed in leadership and general elections manifested itself in a hyperactive commitment to every progressive cause going.

While Corbyn enthused crowds in the 2015 leadership election, McDonnell oversaw messaging and strategy; in the early chaotic days after Corbyn’s triumph, McDonnell acted as a stabilising force. But if there is a political dictum that however close a friendship, tensions between the offices of leader and shadow chancellor will test it, Corbyn and McDonnell definitively prove it. Tensions over a strategy to confront the left’s antisemitic fringe and on Labour’s Brexit position caused genuine strain – and the shadow chancellor is far more willing to compromise with the party’s right than his old reputation as the “hard man of Corbynism” suggests. In truth, McDonnell scents a historic opportunity to enact the radical economic agenda he has spent decades formulating, and fears it being unnecessarily snatched away because of self-inflicted injury. Shifting Labour’s position on Brexit to accommodate remainers, for example, was something his team saw as an issue that threatened the entire project.

Which brings us to the departure of Karie Murphy from the management of Corbyn’s office. Corbyn could hardly be further from the hostile stereotype of a demagogic leftist enamoured of confrontation. The leader is conflict-averse, which is not something that can be said of Murphy, a former nurse and office manager for Tom Watson before their bitter schism over Corbyn’s leadership. When she assumed her role in 2016, she imposed discipline and order on a chaotic operation, was pivotal in heading off the coup attempt, and drove through a flagship community organising initiative.

But Murphy undoubtedly eclipsed McDonnell’s old role, becoming the second most powerful figure in the leader’s office after Corbyn himself. The rise of powerful officials within the Corbyn project led one senior figure to half-jokingly suggest reading Leon Trotsky’s work on how the Soviet bureaucracy subverted the Russian revolution. There is perhaps a more measured way to describe Murphy’s time in the leader’s office: without her efforts to professionalise, Labour would not have achieved its 2017 result, but the same stubbornness and determination that saved the Corbyn project stopped it becoming flexible in a period of unprecedented political tumult. Andrew Fisher’s announcement that he was leaving at the end of the year caused genuine internal shockwaves, too, caused not by Murphy’s management style – unlike some of his colleagues, the two did not clash – but rather general frustrations with the operation.

What has undeniably reasserted itself is the Corbyn-McDonnell axis. Crucially, this has been at the instigation of Corbyn himself: the bungled attempt to remove Watson as deputy leader provoked genuine rage in the Labour leader and led him to conclude that sweeping internal change was necessary. This was a position his two longest-standing allies, McDonnell and the shadow home secretary, Diane Abbott, had long held.

The challenges now confronting the project are formidable. A Labour opposition office is a hothouse at the best of times, but for those who expected to be consigned to the fringes of politics, being plunged into the heart of Britain’s greatest political tumult since the second world war has caused huge personal stress. The Brexit culture war has eclipsed Labour’s domestic agenda, riven havoc amongst its 2017 electoral coalition, and caused profound internal splits. Some senior MPs representing urban constituencies talk of furious remainers currently showing no interest in returning to the Labour fold; an MP representing a leave community complains that ultra-remainers show no interest in constituencies such as their own. While Corbyn is often caricatured as a closet lexiteer, Brexit is, in truth, a demoralising distraction from the things he’s interested in: aides speak of his eyes lighting up when ideas to address the climate emergency are mentioned.

The operation appeared blindsided by Boris Johnson’s rise to power: its leading figures are keenly aware that his key messages, such as “get Brexit done” are clear and have cut through. Labour risks being defined by its Tory opponents, and core messages from the party have yet to be signed off. Yet amid the angst, there is still genuine optimism to be found. However difficult the circumstances now, senior Labour officials remark, things were far worse before the 2017 election – Theresa May’s ill-fated decision to call it saved a Corbyn project that otherwise could have collapsed within a year.

The party’s domestic agenda – which will be sharper and more radical than in 2017 – is demonstrably popular, but only an election provides a real opportunity for it to cut through. A mass membership will be mobilised, Labour will clearly offer the best realistic route for remainers to stay in the EU, and as the 2017 election showed, the more exposure Corbyn receives, the more he is liked. It is when campaigning that Corbyn is in his element. But as this week’s British Election Study underlined – the research showed party loyalties have been significantly eroded – we live in a time of unprecedented political fluidity. Any confident prediction of the coming contest emerges from either naivety or delusion.

• Owen Jones is a Guardian columnist