There’s a terrific scene in The Favourite, the film about the last of the Stuart monarchs, when Queen Anne is about to propose a European policy to her parliament. Just before she opens her mouth, the bewigged MPs make plain that they won’t have it. The queen faints with shock and hits the floor with a thump.

I don’t expect Queen Theresa to collapse when she loses in parliament this week, not least because the rejection of her Brexit deal is not going to be a surprise to anyone, herself included. If we still lived by the traditional rules of British politics, defeat ought to be the final curtain for Mrs May. Brexit is the defining task of her premiership and she is about to fail it. We’d ordinarily expect the abdication of the PM to follow such a humiliating rejection. Yet no one, neither friend nor foe, expects her to respond to defeat by submitting her resignation. Brexit has so scrambled our politics that it has normalised dramas that we would once have regarded as extraordinary and made the unthinkable routine. We no longer know which of the rules still apply.

Some illumination of how Britain and its parliament got trapped in this howling nightmare is to be found in The Favourite. I hugely recommend this film and not just for the dazzling performances by Olivia Colman, Rachel Weisz and Emma Stone. The drama is footloose with the facts, but it is a prompt to pay more attention to a critical period of our history that is little taught in schools and little known to most Britons. The tensely convoluted relationship between parliament, ministers, monarch and favourites, played out during a war in Europe, echoes into our own era from the age of Anne. Her grandfather, Charles I, had been deprived of his crown and his head after he embarked on a losing struggle with parliament.

When her father, James II, exhibited designs to create an absolutist Catholic monarchy, parliament rebelled again. He lost his crown, but escaped with his head, in the Glorious Revolution of 1688. By Anne’s time, quite a lot of clout was still wielded by the monarch – or by those, such as John and Sarah Churchill, who could get close enough to the monarch to harness the powers of the crown to their own ambitions. The queen could still dismiss her ministers, but power was leaching away. Anne was the last British sovereign to veto a parliamentary bill.

Britain had begun its gradual transformation, centuries in the making, into a liberal democracy with an ornamental constitutional monarchy. One consequence of getting there by evolution rather than revolution is that Britain has never had a properly codified constitution. Other nations have sat down, thought about it and inscribed the rules in famous documents with resonant phrases. We have made things up as we have gone along, with the result that many of the laws governing our politics are ambiguous and contested. We never had the equivalent of the founding fathers of the United States. Unlike postwar Japan and Germany, Britain never suffered a devastating military defeat and then had a constitution imposed on us by the winners.

You can argue that an ad-hoc constitution has not served Britain entirely badly. We have bolted on innovations in recent decades, such as the Scottish parliament, the Welsh assembly and the supreme court, with a relative lack of turbulence. Gaps in the constitution in protecting rights have been filled by membership of the European Union. The previous disinclination of Britons to go to war with themselves over constitutional issues has also meant a high tolerance for absurdities, such as a bloated, unelected second chamber that still calls itself the House of Lords and still contains hereditary peers.

‘The British way of doing politics is... expressed in the very architecture of the chamber of the House of Commons.’ Photograph: HANDOUT/AFP/Getty Images

The constitution is stuck together from parliamentary conventions, precedents, international agreements, unwritten understandings, judicial rulings and legislative sticky notes. This seemed serviceable enough – until this spatchcocked structure collided with something as colossal as Brexit. We are partly paying the price for making such a massive decision by simple plebiscite, without having properly settled rules about referendums and how they can be reconciled with representative democracy. It is very hard work to amend the constitution of the United States and a change can only be made if there is a wide and deep consensus. Britain is heading out of the EU, the most consequential act in decades, on the basis of one ballot held nearly three years ago in which just one vote could have decided the outcome. The closeness of the result and the lack of agreement about what it meant spelled trouble from the start. It also led to the inevitable convulsions that were going to follow when a parliament largely against Brexit was tasked with implementing a result most MPs thought a mistake, a challenge without precedent. This conundrum might have been eased had Mrs May responded to her task with a non-partisan, cross-party approach, but she made things harder when she started out by seeking to please one faction of her party alone.

The conventions and culture of parliament have deepened the nightmare. The British way of doing politics is founded in the idea that power is a binary contest between two big and tribal parties. It is expressed in the very architecture of the chamber of the House of Commons that sits the two sides confronting each other two swords’ length apart. It is incarnated in parliamentary rules that vest a large amount of power in the two tribal leaders – the prime minister and the leader of the official opposition. To compound the problem, it is a hung parliament in which the prime minister is a former Remainer trying to find a form of Brexit that can satisfy a majority and the opposition leader is a Brexiter leading a Remainer party who has little interest in trying to resolve the deadlock and lots of incentives to want it to end disastrously. The two of them have great sway over how parliamentary time is allocated and which motions MPs get to debate. This gives them the power to run down the clock – and both have been exploiting this for different reasons. Mrs May has taken Britain perilously close to the precipice of a crash-out Brexit on the gamble that, when MPs are staring into the abyss, they will finally succumb to her deal. This can be fairly called Blackmail Brexit. Even if she does ultimately prevail this way, forcing MPs to approve a plan they hate under the muzzle of the gun will guarantee further trouble and strife.

Parliamentary convention has it that only the leader of the official opposition can table a motion of no confidence in the government. This has given Mr Corbyn an extremely convenient hiding place from his own contradictions. He keeps calling for an election, but repeatedly refuses to trigger the one mechanism that could make that happen because this delays the day when he has to declare whether he is or is not in favour of another referendum. If he fails to table a confidence motion this week, it will become too late to hold an election before Britain is due to leave the EU.

In recent days, we have seen senior MPs from different parties working together to try to navigate an escape from the impasse. There is an embryonic cross-party coalition for seeking a solution, but it is desperately late and the obstacles to coalition-building, both cultural and mechanical, are huge. They have had to resort to parliamentary devices that will seem bewildering and arcane even to the most intelligent and engaged of voters. There was a great fuss when the Speaker allowed an amendment, which subsequently passed, instructing Mrs May to quickly report back to parliament when her deal is voted down. We are witnessing these parliamentary explosions about procedural issues and sequencing questions because MPs lack the tools to properly take control of Brexit decision-making.

People I respect think that when the dust has finally settled, Britain will need to rethink its casual attitude to the rules of its democracy and embrace a properly codified and protected constitution. Professor Vernon Bogdanor will make that case in a book, Beyond Brexit, to be published in February.

Britons might have avoided this nightmare – or at least been better prepared to cope with it – had we understood more of our history. Queen Anne, who lost all 17 of her children, died without a direct heir. Rather than risk giving the throne to another troublesome Stuart, pragmatic British parliamentarians recruited a replacement monarch from Germany who did not speak a word of English. His great-great-great-great-great-great-great granddaughter now wears the crown.

• Andrew Rawnsley is an Observer columnist