There comes a point in every party – as dawn breaks, and the hangovers are kicking in – when the last revellers realise that anyone still on the premises risks being stuck with the job of clearing up the mess. Standing now in the rancid-smelling Brexit kitchen, Dominic Raab has realised that he does not want to be that person – and he’s out. Esther McVey, work and pensions secretary, has followed him through the door.

Every resignation is damaging to Theresa May, but Raab’s is doubly so. First, it means that there is now no Brexit true believer prepared to take an author’s credit on the deal that is about to come before parliament. Yes, there are still a few class-of-2016 leavers in cabinet – Michael Gove, Liam Fox, Andrea Leadsom and others – but they have just been cheerleaders, and pretty muted ones at that. It isn’t clear at the time of writing how long they will stick around, offering a bit of tepid golf applause at the margins. But Raab was in a different category. He was in the room where Brexit happened. Notionally, at least, this was his deal. (In reality, it was all hammered out by officials in direct communication with the prime minister, which was a slow-burn injury to the Brexit secretary’s pride.)

These resignations confirm a fundamental structural problem with the whole leave prospectus: it was a fantasy, and as such incompatible with the mundane fulfilment of ministerial responsibility. Raab has come to the same conclusion that David Davis and Boris Johnson reached earlier in the year: it is easier to be on the team that accuses the prime minister of failing to deliver majestic herds of unicorns than it is to be stuck with a portfolio that requires expertise in unicorn-breeding.

The loss from cabinet of the man whose job title literally implied ownership of the whole process also signals to the country that there is something intrinsically dysfunctional about the project – both May’s government and the deal it is peddling.

Remainers will see it as further proof that the flaw lies in Brexit as an enterprise. The job is being done badly because it was a bad idea to begin with. There was no available model for doing it well. Leavers will find confirmation that the spirit of their ambition has been betrayed by a prime minister who never truly internalised the vision. How, they might reasonably ask, can Brexit be implemented by people who deep down never really thought it was a good idea in the first place? To which the pro-European response is another question: how is it that so many of those who really thought it was such a good idea have fled responsibility for its practical application?

And there lies the second way in which Raab’s departure wounds the prime minister. It collapses one of the bridges along which other MPs might have walked, albeit reluctantly, towards supporting May’s compromises. She needs a critical mass of Tories prepared to line up with the cabinet, stand in front of a TV camera, conjure the twin horrors of no deal and no Brexit, and invoke duty to the national interest in endorsing what is advertised as the only deal on offer. Every conspicuous refusal to be part of that show has a knock-on effect disincentivising waverers with one eye on history’s verdict and another on their career prospects. Already Suella Braverman, a junior minister in Raab’s department, has followed her former boss. No one wants to be the mug who gambled everything buying shares in the prime minister when the rest of the market is selling like mad.

Brexit has been a wild ride so far, exhilarating at first for those who wanted it, terrifying for those that didn’t. But the party is drawing to a close. The music has stopped. The place is strewn with cigarette butts and empty beer cans. The black bin bags are being passed around. The faces still in the room are wan and haggard. They know that there is nothing left to do but survey the damage, count the cost. Is it any surprise that the last wreckers in cabinet are making excuses and heading for the door?

• Rafael Behr is a Guardian columnist