Everything that could be done had been done to secure a victory for Boris Johnson on what had been billed as “Super Saturday”. He had come back from Brussels with his so-called “new deal” to the sound of the massed brass bands of the pro-Brexit media trumpeting praise for a “stunning achievement”/“personal triumph”/insert more sycophantic hyperbole here. Some European leaders tried to help him frame the choice before parliament as “new deal or no deal”. Suggestions that the EU might not grant another extension to the withdrawal date were designed to spook MPs into voting yes for fear of hurling Britain into a crash-out Brexit.

Cabinet ministers were deployed to “man-mark” any Conservative MP whose vote was doubtful. Heavy-breathing Tory whips said they were going to get “medieval” with rebels. The self-styled Tory Spartans, who would have spat out the Johnson deal as treachery if it had been presented by Theresa May, had fallen into line. Some of them had begun to see the ridiculousness of being Brexiters who never actually vote for Brexit.

Then there was the timetabling. To further ratchet up the pressure, the government staged the crackling drama of an “emergency” Saturday sitting of parliament, the first time that MPs had met at the weekend for nearly 40 years. This meant that everyone had an absurdly limited amount of time to get their heads around the latest tortuous iteration of Brexit. Concluding yesterday’s debate on behalf of the government, Michael Gove declared: “Our democracy is precious and this parliament is a special place.” They had a funny way of showing this supposed reverence for democracy and parliament. MPs were being asked to make a decision with huge consequences less than 48 hours after the deal had been unveiled. Were you able to conduct a confidential survey, guaranteeing to parliamentarians that their responses would remain anonymous, it is my strong suspicion that well under half of them have actually read the legal text and the rewritten political declaration. The government’s desperation to stampede parliament into signing off on the deal was further illustrated by its point-blank refusal to publish any analysis of its economic impact.

Facebook Twitter Pinterest Oliver Letwin greets supporters in Whitehall after his amendment succeeded in the Commons. Photograph: Simon Dawson/Reuters

These tactics ultimately backfired. There was too much resistance to the attempt to bounce MPs into agreement. The debate was peppered with complaints that a prime minister who couldn’t be trusted was seeking a blank cheque from a parliament being kept in the dark. Justine Greening, the former Tory cabinet minister, tellingly complained that this was like being asked to buy a house without being allowed to see it.

By 322 to 306, a 16-vote defeat for Boris Johnson, MPs thwarted him by backing Sir Oliver Letwin’s amendment to withhold approval of the deal until MPs have had the chance to properly scrutinise the withdrawal legislation. For the prime minister, so-called Super Saturday turned into Squelched Saturday.

To understand why he was defeated, you first have to consider his deal. It bears no resemblance to the have-your-cake-and-eat-it promises peddled to the country by him and his fellow travellers during the referendum campaign in 2016. He succeeded in getting rid of the “backstop”, the element of the old deal so aggravating to the Tory Brexit ultras, but at the cost of inserting a frontstop that will keep Northern Ireland largely aligned with the single market and customs union for at least five years. The creation of an economic border down the Irish Sea made it impossible for him to gain the support of the Democratic Unionists. The other major difference with Mrs May’s deal is that it envisages moving to a much more distant relationship with Britain’s most important trading partners. The Johnson deal is one of the rock-hardest forms of Brexit. That diminished his chances of attracting support from Labour MPs.

Play Video 1:32 Saturday’s Brexit vote in 90 seconds – video

In his speech to the Commons yesterday, the Tory leader said it was time to “move on”, a mantra parroted ad nauseam from the benches behind him. But as some of the more clued-up MPs observed, his deal does not “get Brexit done” at all. It covers only the divorce and a period of “transition”. Where Britain ultimately lands is still hugely uncertain. It is merely the prelude to a tougher stretch of bargaining about the terms of trade, customs, tariffs and standards with the EU. These negotiations come with another deadline attached. The cliff edge moves to the end of 2020. Caroline Lucas, the Green MP, put it wittily when she remarked that it was “a bad deal with a backdoor to a no deal.”

The fundamental trouble with this deal is the same as with all the many versions of Brexit floated by two Tory prime minsters over the past three-plus years. None offers terms as favourable to the United Kingdom as remaining within the European Union. A study by the UK in a Changing Europe thinktank projects that the Johnson deal will shave up to 7% from the per capita income of Britons over a decade. Other forecasts are available. None of the credible ones suggests that Britain will be better off outside the EU.

There were other reasons why Mr Johnson could not assemble the coalition of support that he needed. The 10 votes of the Democratic Unionists played a decisive role in his defeat. They radiated the fury of people betrayed. Mr Johnson attended their party conference last year to pledge that “no British Conservative government could or should sign up to” regulatory checks and customs controls between Northern Ireland and the rest of the UK. Now he is pushing a deal that does this very thing.

Facebook Twitter Pinterest ‘Justine Greening, the former Tory cabinet minister, tellingly complained that this was like being asked to buy a house without being allowed to see it.’ Photograph: Daniel Leal-Olivas/AFP via Getty Images

Without the DUP, it was always going to be a very hard scrabble for the government to command a majority in the Commons. Its hopes depended on convincing enough Labour MPs to vote with the Tories. There is a band of Labour MPs who have their reasons for wanting to “get Brexit done”, but most had even more compelling reasons for not helping Mr Johnson to do it. He gave them less incentive to support him and more reason to distrust him by moving assurances on workers’ rights and environmental standards out of the binding withdrawal agreement and into the non-binding political declaration. His behaviour in his 88 febrile days as prime minister made it yet harder for Labour MPs to lend him their support.

The unlawful prorogation of parliament and the use of incendiary language to attack parliamentarians alienated some of the very Labour MPs he needed to persuade. The fact that it was a Johnson deal was a very big problem for them. The intense loathing he arouses among Labour people is much more visceral than their feelings about Theresa May. It is a very big step for a Labour MP to enable a Tory prime minister, especially when an election is looming. When that Tory prime minister is Boris Johnson, it proved just too much.

'Those hoping to take the question back to the people have more time to convince parliament to embrace a fresh referendum.

In response to his defeat, the prime minister rose to the dispatch box, fixed a smile to his face and spoke as if, to use a phrase made notorious by his predecessor, “nothing has changed”. Of course, quite a lot has changed. By the time you read this, the government will have sent a letter to the EU requesting an extension to the Brexit deadline, something Mr Johnson has repeatedly sworn he would never do, or he will be in breach of the law.

His deal is not necessarily dead. He mustered 306 votes, 20 more than Mrs May ever got for her deal. This leaves him 14 short of what he needs for a majority. Some of the MPs who defied him over the Letwin amendment, including Sir Oliver himself, have said they will support the government when it comes to votes on the Brexit legislation. There’s not much doubt, though, that the road ahead has become a great deal more rocky for Mr Johnson. Parliament will be able to seize the opportunity to subject his deal to the searching and detailed scrutiny that the government sought to evade yesterday. Pressure can be increased on ministers to reveal the true costs of the Johnson deal. Those hoping to take the question back to the people have more time to convince parliament to embrace a fresh referendum.

Boris Johnson called this special Saturday sitting in the hope that it would give him a reputation-boosting, momentum-building victory to flourish. He wanted to be able to claim that Brexit was done and dusted. Instead, Brexit is not done and he is dusted.

Much is in flux after another “historic” parliamentary vote that failed to settle Britain’s future. One thing is certain. Our long national nightmare continues.

• Andrew Rawnsley is Chief Political Commentator of the Observer