On Wednesday, in her last major speech, Theresa May lamented a growing culture of absolutism in politics. Introspection was never her strong suit, and this speech to the Chatham House thinktank was no exception. In a scarcely veiled swipe at the populist right, at home and abroad, she bemoaned an “inability to combine principles with pragmatism … [from those who believe] that if you simply assert your view loud enough and long enough you will get your way in the end”.

This from the leader who produced a deal without consulting other parties, refused to compromise on it, put it to parliament three times and lost every time, and would have tabled it for a fourth time if she’d been able.

The kipper, intended as a symbol of EU overregulation, actually showcased Johnson’s lack of attention to detail

On the same night her two potential successors, Boris Johnson and Jeremy Hunt, presented themselves to her party at a hustings one last time. Johnson, the favourite, came on stage waving a kipper, given to him by the editor of a national newspaper. “[He] received it from a kipper smoker in the Isle of Man,” explained Johnson, “who is utterly furious because after decades of sending kippers like this through the post he has had his costs massively increased by Brussels bureaucrats, who have insisted that each kipper must be accompanied by a – this – a plastic ice pillow!”

The kipper, intended as a symbol of EU overregulation, actually showcased Johnson’s legendary lack of attention to detail. The Isle of Man is not in the EU. Were he right, the story would really illustrate what life will be like when we leave and have to conform to EU regulations but have no say in making them. But he wasn’t. The new packaging rule is actually a UK one.

And so it was that one ineffectual Tory leader prepared to leave the stage while another, more theatrical but even less credible, got set to enter it.

Since she lost her Commons majority during the 2017 snap election, May has been living on borrowed time. That time is now up. Next Wednesday she’ll be at the despatch box for her last prime minister’s questions. By that night she’ll be gone.

Her shortcomings are many and her failures in office were self-evident. To remember them is important. To dwell on them seems almost ghoulish. She deserves no pity. When Sylvester Marshall, who lived in Britain for 44 years, was denied cancer treatment on the NHS after being ensnared in the Windrush scandal (May’s scandal), she told him he needed to “evidence his settled status”. The devastating appraisals of her tenure – and it’s difficult to imagine what other kind there might be – should be understood less as cruelty than as karma.

But the question of how someone so inept, ill-suited and ill at ease, with so little charm, authority, charisma or capacity, could stay in that job for as long as she did deserves interrogation.

May was hopeless. But she was not useless. Her party needed her. When first elected leader, the Conservatives hoped that she might emulate Thatcher and secure a reasonable deal through steely resolve. When she called the snap election they hailed her as a genius for having the gumption to exploit Labour’s weaknesses and expand her majority.

Although it is difficult to imagine now, support for her at times could be effusive. She was regarded as a real asset to her party. “This former lukewarm remainer has truly been the servant of a newly sovereign people,” wrote Dean Godson, director of the rightwing thinktank Policy Exchange, during what is now considered one of the worst election campaigns in living memory. “Her tone has captured the moment … Her own form of provincial respectability is now aligned to the needs of those ‘people from somewhere’ who voted leave.”

Play Video 1:20 Theresa May refuses to criticise Johnson's no-deal Brexit claims – video

The media class, convinced that Labour under Jeremy Corbyn was destined to fail, were complicit in seeing attributes in May that did not exist. The closer they were to power, it seems, the less they could actually see. At 9.53pm on election night Piers Morgan tweeted: “As exit poll looms, I repeat my prediction: Conservatives to win by 90-100 seat majority.” At 9.57pm Steve Hawkes, deputy political editor of the Sun, tweeted: “Rumour Tories could be looking at 400 seats.”

At 10pm we learned that she would likely lose her majority altogether. Even then, though, she was useful to the Tories as a placeholder – keeping the seat warm while the party found someone more plausible while making sure that Corbyn did not occupy it. It is worth remembering that they did not get rid of her – nobody more plausible did come along. She quit.

May was the best candidate for the job. The alternative was Andrea Leadsom, only recently aware that climate crisis was real, who has almost always voted against laws to promote equality and human rights and thinks that issues of Islamophobia in the Tory party should be taken up by the Foreign Office.

May has done a better job than her predecessor, David Cameron. That’s not saying much.

Where legislation is concerned she achieved little while she was premier because Brexit took over everything even if it didn’t go anywhere. That’s a problem because an awful lot needs to be done, and a lot of what has been done needs to be reversed.

Cameron on the other hand made things far worse – hiking tuition fees, slashing benefits, pursuing austerity and presiding over an exponential increase in the number of food banks and of people using them.

May could not navigate her way out of the Brexit mess. But Cameron created it. Calling a referendum, losing it and then fleeing the scene almost immediately.

Her successor is likely to be worse too. Johnson is a better communicator; but chutzpah is not a strategy and pomposity is not a plan. He was both a disaster and a liability as a foreign secretary and is not taken seriously abroad. He has neither a viable plan for securing a different deal than May’s nor for how to proceed should there be no deal. His contribution, in this critical moment, is wordplay and a joke. “Whatever deal we do there will be adequate supplies of glucose,” he told the crowd on Wednesday evening. “And milk solids and whey to make the Mars bars that we need, because where there’s a will there’s a whey.”

The crowd cheered. But the EU will probably have the last laugh because the deal May came back with was in all likelihood the best deal a Tory could get. This was not the failure of an individual but of a party and a system. She was the best they could do and she did the best she could. This is the problem. This is why we need an election.

• Gary Younge is a Guardian columnist