The collapse of the Tory-Labour Brexit talks surprised Westminster as much as did the news that bears conduct their ablutions in forested areas and Boris Johnson is interested in becoming prime minister. A question worth investigating, because it illuminates where we are now, how we got here and where we are heading, is whether that failure was inevitable. Was this bizarre and ultimately fruitless chapter of the Brexit saga never anything but a protracted charade? Was the inability to broker a compromise always written in the stars? Or could different actors have found a way to release Britain from its long national nightmare?

Many believe that these talks, the last resort of an enfeebled prime minister, were fated to fail. One senior figure involved in the six weeks of negotiations speaks as if describing a doomed relationship: “It was always going to end this way.”

Neither side acted from the purest of motives. For Mrs May, this gambit was a desperate roll of the die after repeated defeats in parliament at the hands of her own Brexit ultras. She hoped to buy herself extra time at Number 10 and have something to take to the EU to justify the extension to the withdrawal date. Labour’s main motive for participation was revealed by its spokespeople’s repeated mantra that they had entered the negotiations “in good faith”. They wanted to look like people prepared to make an effort to end the paralysis for fear of being otherwise painted as a cynical gang simply hoping to make an electoral profit from the misery.

The personalities of Theresa May and Jeremy Corbyn, neither of whom is famous as a consensus-seeker, had many doubting that they could ever reach common ground. When they met, others in the room reported that they addressed each other as if they were representatives of alien species from the opposite ends of the galaxy. Yet it is just about conceivable that, left to themselves, they might have struck some kind of deal. Their differences over the withdrawal agreement were not all that great. Mrs May even seemed prepared to embed protection of workers’ rights in legislation, one of the Brexit issues that the Labour leader does care about. She had motive to secure a deal so that she could depart Number 10 with a few shreds of dignity and some kind of legacy.

He had some incentive to resolve the parliamentary deadlock. The Labour leader’s ill-disguised secret is that he wants to move the national conversation off Brexit, where his own views run counter to most of his party, and on to themes such as austerity and inequality where he is much happier and thinks a Labour election victory can be found. Had the two leaders been locked in a room together, and told they didn’t have to worry what anyone else thought, I can almost imagine that they might have struck some kind of bargain. What prevented that was their respective parties. Reading the last rites over the talks, Mr Corbyn blamed the impossibility of negotiating with a Conservative party in such an advanced state of meltdown. In response, Mrs May blamed divisions within Labour between those who want to deliver Brexit and those who seek to reverse it. Both had a point.

A Brexit protest outside the Houses of Parliament on 14 May. Photograph: Amer Ghazzal/Barcroft Media

One of the biggest obstacles to any compromise was Mrs May’s lack of authority over her warring tribe. Her position was too fragile to be able to guarantee that she could deliver enough Tory MPs behind any deal to make one viable. The talks were regularly punctuated by members of the cabinet attempting to torpedo them by issuing ultimatums. As it became clear that Mrs May had entered the twilight zone of her premiership, the Labour side was increasingly concerned with how any deal could be made “future-proof” against a new Tory leader. Some of the Conservative figures involved in the talks were very open to that idea. Philip Hammond and David Lidington have reason to be anxious that they will not survive a change of prime minister. Yet it was impossible to find a way to bind whoever succeeds Mrs May to any agreement that she might make. This held out the prospect that the Tory leadership contest would ring to the sound of candidates saying they would destroy any compromise. A death knell for the talks was the public letter signed by Tory wannabe leaders threatening to rip up any agreement.

Some involved, both on the Labour side and the Tory, couldn’t help but think that much might have been different had the two parties engaged with each other earlier, rather than meet at one minute to midnight when Mrs May had already gone down to a series of epic defeats in the Commons. Whether she should have looked for a compromise much earlier will be a question that will detain historians. Their judgment will shape a lot of the verdict on Mrs May’s tortured premiership.

She has been a member of the Conservative party all her adult life and anyone familiar with the party’s behaviour over the past three decades might have intuited that it would be impossible to unite Tories around any Brexit strategy. A leader with more foresight than Mrs May could have worked that out and realised that the only way to manage it through the Commons was by building a cross-party majority. Had Mrs May begun Brexit by going to Brussels with a negotiating mandate pre-approved by parliament, she might have enhanced her clout with the EU as well as her chances of securing parliamentary agreement on the outcome.

Fatefully, she made a very different choice. Instead of trying to build a parliamentary consensus and seeking common ground between the 52% and the 48%, she chose to entrench divisions within the Commons and inflame them in the country by taking one side against the other. When she failed to get the mandate for a hard Brexit that she sought at the 2017 election, she might have switched strategy and attempted to reach out to the opposition at that point. She instead doubled down on her original, fatal strategy and made herself a hostage of the DUP and the Tory ultras on her backbenches.

Everyone involved in the Tory-Labour talks knew that any kind of compromise would leave vast numbers of voters hugely unhappy.

This is not just because Mrs May is a rigid and unimaginative character. She is the product of a political culture that tends to emphasise the adversarial over the consensual. It is expressed in the architecture of a parliament that sits the two sides confronting each other. European countries with more experience of coalition governments are schooled in the art of the compromise. The idea is foreign to the winner-takes-all tradition of British politics. This is especially true of a Tory party that reveres Margaret Thatcher above all its other leaders since Winston Churchill. The Conservative party is in love with the concept of the battling leader and rather disdains the idea of the healing leader.

The collapse of these talks is of a piece with the narrative arc since the referendum in the summer of 2016. For nearly three years, the scope for compromise has steadily shrunk as opinion on both sides has become more radicalised. Advocates of Brexit who once said they’d be content to be out of the EU but stay within the single market have since transmogrified into adamantine no-dealers. Remainers who might once have settled for a halfway house will now accept nothing other than a second referendum. Everyone involved in the Tory-Labour talks knew that any kind of compromise would leave vast numbers of voters hugely unhappy: another factor dooming the effort to failure. For many Brexiters, any outcome which is not the most purist version of that enterprise will now be seen as a betrayal. For many Remainers, any result other than the reversal of Brexit will be intolerable.

This polarisation will be magnified by a strong showing by the Brexit party in this week’s European elections. The looming Tory leadership contest will drag the contenders, even the more sensible ones, further towards the most extreme forms of Brexit as they compete for the support of the party’s anti-deal members.

Pressure is building within Labour for the party to take an unambiguous stand on the other side of the barricades and become an anti-Brexit party. That pressure will be increased when the Euros see large numbers of previous Labour voters desert the party for the Lib Dems, Greens and Change UK. If a general election hasn’t happened by September, Labour’s party conference is highly likely to force its reluctant leadership to make a no-qualifications commitment to a fresh referendum. The middle ground, such as it was, has become scorched earth. The chances of this concluding with no Brexit or a no-deal Brexit are both rising sharply.

• Andrew Rawnsley is Associate Editor and Chief Political Commentator of the Observer