It is late October in the House of Lords and the chamber is full. The peers squeeze on to benches, or crowd four deep around the throne. The government has threatened to suspend the Lords, which is about to debate the abolition of tax credits. It is not the first time this House, which was last suspended by Oliver Cromwell, has faced an existential threat. You could say it is part of its remit.

The chamber was designed by Augustus Pugin who, shortly afterwards, went mad. It is Hammer Horror red. The Lord Speaker is on her woolsack, with shoes dangling; the clerks are at their desk, wearing wigs and holding iPads; the bishops are on their bench, which is the only one with arms. A forest of microphones hangs from the gilded ceiling. Hogwarts, everyone calls the Lords, but on a flood plain and fraying. Outside the chamber, cupboards stand open and bulge with wires. Ladders lead to locked trapdoors, or nowhere.

No one has to attend the Lords, and no one is paid a salary for doing so (for those who turn up, there is an allowance of £300 a day, expenses not included). But the Tories are panicked, and have summoned “backwoodsmen” – those who rarely come. I can see Andrew Lloyd Webber and Julian Fellowes on the Tory benches; the relics of Thatcherism, like the prop room of Spitting Image; the new female bishop of Gloucester is among the Lords Spiritual. Melvyn Bragg, Joan Bakewell and the Kinnocks are across the floor.

Labour member Patricia Hollis, who led the tax credit revolt. Photograph: Ben Quinton/The Guardian

The abolition of tax credits, which the prime minister said would not happen during last year’s general election campaign, did not have to come before this House. Since the Lords threw out the Liberal budget in 1909, there has been a convention that they do not interfere on financial matters. But George Osborne, seeking to avoid growing opposition from his own side in the Commons, has placed the change inside secondary legislation – a statutory instrument, or SI. This means it can be rejected or delayed, but not amended. And, since the original tax credits legislation was not deemed a “money bill”, the SI to amend it isn’t subject to financial privilege. It is a major error by the chancellor. The Conservatives are not the largest party (they lost their dominance when the hereditary peers left in 1999). Patricia Hollis, for Labour, has tabled a motion that would force the Commons to delay. The Tories, though, call it a “fatal motion”, a move to kill the legislation completely – not a delay at all.

The leader of the House rises. Her name is Tina Stowell. She is small and neat. Her nickname here is “Tina the typist”, because she used to be William Hague’s deputy chief of staff. (It’s snobbery, yes, but I don’t think anyone could accuse Stowell of dangerous charisma.) She insists that the regulations before the House are “part of our wider economic strategy and vision for the future”. She says she has been to see Osborne. He will listen carefully to the Lords (and here is the threat), if they express their concern “in the way that it is precedented for us to do so”. That is, he will listen only if he is not compelled to do so.

This debate, then, is about whether the Lords have the authority to overrule the Commons. The answer is: they do. But Osborne will not admit to a mistake, so the Tory line must be: although you are right on a technicality, you are, more generally, wrong, because this is a budgetary matter. Tory peers rise, one after the other, to insist on this. “I was for six years the secretary of state for health and social security and, as such, no one’s idea of a natural supporter of the Treasury and all their schemes,” says Norman Fowler. Nigel Lawson murmurs, “You can say that again.”

The debate is grandiose and self-consciously polite, as if the peers cannot forget that every word will be transcribed into Hansard, where it will remain after everyone is dead. There is none of the shrieking anti-intellectualism of the House of Commons: 500 men auditioning to lead News At Ten.

A board in the Prince’s Chamber where phone messages are written down and left for members. Photograph: Ben Quinton/The Guardian

Molly Meacher, a crossbencher, frets that she is “acutely conscious of the threats made by the government to destroy this House”. She does not enjoy this kind of pressure, she says. David Cameron has threatened to fill the House with 100 Tory peers, so that it will be the biggest party again, as it has been for 107 of the last 115 years. Even Tony Blair, in the maddest days of his Messianism, did not threaten that. If Blair created more peers than any prime minister in history (35.7 a year), Cameron is producing them at an even higher rate: 47 a year, or 237 so far.

Hollis, who won the Orwell book prize and speaks like an immensely skilled psychotherapist, asks why the Lords should treat this as an off-limits budgetary matter, “when the government, who could have made it so, chose not to do so?” The so-called constitutional crisis is “a fig leaf possibly disguising tensions in the Commons between members of the government”. She talks about the families who will receive letters “at Christmas”, telling them they will lose tax credits. She speaks for 13 minutes and is heard without interruption. Osborne, here, is Ebenezer Scrooge without the third act redemption; David Cameron is a man who lied to the electorate, and then claimed a mandate to make the changes he said he would not make.

Hollis’s amendment is passed by 289 to 272: the Liberal Democrats, 93% of whose parliamentary party is now in the Lords, vote with Labour. Not one Conservative supports it, although Hollis tells me later that one in three Conservatives said to her, en route to vote the other way, “Sorry about this”, or “Super speech”, or, “I hope you win!”

The Royal Gallery looking towards the Prince’s Chamber. Photograph: Ben Quinton/The Guardian

A month later, Osborne changes his mind on tax credits, as if he has taken the Lords’ advice and the House has, therefore, perfectly discharged its theoretical function to scrutinise the executive. But the day after the debate, the government also commissions a review into the Lords’ powers: Lord Strathclyde, a former Tory leader of the House, is asked to write it. The Tories do not want a nest of opposition, where Labour can combine with Liberal Democrats and crossbenchers to thwart them. Control freaks: every government is full of them, but this is something new.

***

The House of Lords is the second largest legislature in the world after the National People’s Congress in China. Never elected, and reformed only piecemeal, it is also the oddest. The ladies toilets are green marble, and contain a red leather chaise longue. The doorkeepers wear white tie, like pianists. Nigel Lawson does not wear a coat, but an opera cloak. Black Rod, who is in charge of security, wears an ornamental ruffle at his throat.

The Lords has 859 members, and a chamber built to seat 240. The average age is 69. Death used to be the only exit (now you can retire) but, according to Jenny Jones, the Green peer, “you can’t die in parliament. You’re not allowed.” There is a library, a dining room, a tea room, a bar, committee rooms and offices (six peers can share just one office). These offices house the 92 hereditary peers who survived the Blairite cull in 1999, as well as scientists, historians, former civil servants, lawyers, surgeons, party hacks and donors, a dentist, a cheese maker, former MPs, diplomats, a children’s TV presenter and 26 bishops.

If you want to know a place, ask a novelist. So I accost Michael Dobbs, the Conservative life peer who wrote House Of Cards. We sit in the tea room overlooking the Thames. Abolitionists will be thrilled to learn that the coffee is awful.

There are brilliant women here of the kind usually ignored by popular culture: shrewd, committed, post-menopausal

“As you come in, it looks like a kindergarten, doesn’t it?” he asks me. “My natural instinct was to get a giraffe or a teddy to put on my coat hook – to personalise it.” But although the Lords looks like a school, or a gentleman’s club, or the rotting palace of a madman, Dobbs thinks “it is the finest postgraduate institute of learning I have ever attended, and I have attended several”.

He believes the House is the victim of malevolent journalism (he used to be a journalist). He quotes a story that the peers drank 28 bottles of champagne a year each. “It’s absolute bollocks – 95% of that champagne was sold in the shop to tourists.” Dobbs is still searching for the lobster, he says. “I feel very strongly about that, and at times rather bitter. It sounds pathetic, I know, but there is still something called public duty, which a lot of us try to engage in, at some personal discomfort. I’ve got another life, I’ve got a family. You get very irritated at those down moments.”

“We are really parliamentary worms,” he adds. “We take all this sh…” – he corrects himself – “rubbish that comes from the other end of the building [the Commons], and it disappears inside the House of Lords, in darkness, because nobody knows what it is that we do, and it reappears six months later. It might not be perfect, but it is always more fragrant and more fertile than what went in. The House of Commons is really quite disgraceful in the lack of scrutiny. We do all the heavy lifting here.”

Dobbs tells me a story: as a new boy, there was an all-night sitting. They slept in camp beds. He made his bed and went to brush his teeth. When he returned, “some other bastard had pinched my bed and was now fast asleep in it”. Another peer put his teeth on the floor; he never got them back.

***

Back to the fight: I visit the Labour leader, Angela Smith, in her gloomy office. She is friendly and intensely normal; she grew up on a council estate in Essex. She complains about the red robes which – and I am told many times, by peers, to emphasise this – they wear only at the state opening of parliament. “We wouldn’t photograph you in your Halloween costume and then pretend you wear it every day,” is my summary of their complaints. Smith won’t wear ermine. Her robes are “veggie”.

There is a metal clothes rail by the window. “The only wardrobe they’ve got is a Pugin wardrobe,” she says, “and it’s too big to get out the door. I’m not allowed a modern thing.” So she puts her clothes on rails, which are transitory. I will struggle to find a better metaphor for the state of the Lords than Angela Smith’s clothes rail: no one wants it, but they can’t agree on anything else.

The leader of the House, Tina Stowell. Her nickname is ‘Tina the typist’. Photograph: Ben Quinton/The Guardian

Labour’s official policy is for an elected House of Lords, but in the meantime they will work with what there is. In the three months I lurk here, they will drag concessions from the government over the cities and local government devolution bill, the Bank of England bill, the psychoactive substances bill, the charities bill, the enterprise bill, the criminal courts charges regulations and the welfare reform and work bill. Smith thinks the government tried to turn the tax credits vote into a constitutional crisis, “because they didn’t want to talk about the issue”. The Strathclyde review, she says, will be designed “to stop us being naughty again, even when we’re not being naughty”. Her eyes roll around the room: “It’s a bullying tactic.” She thinks it is another manifestation of the government’s desire to neutralise all opposition: from the charities, the unions, the Labour party.

The government, of course, says not. I am, after agonising negotiations, allowed to meet Stowell in her office. Her eyes are warm, and nervous, as she repeats the government line: “The Lords overruled the elected House on a matter of finance. That is wrong.” But she does admit it is part of a wider issue of how the Lords deal with secondary legislation. “The thing we can never forget – and this is so important for the future of the House of Lords – is that the House of Commons has to have the final say.”

Stowell got gay marriage through the Lords; awards from Stonewall and the Pink News are on her mantelpiece. She tells me she explained adultery under the old law to the House like this: “If Waheed Alli were to marry George Clooney and George Clooney had an affair with me, that would be adultery. But if George Clooney ran off with Guy Black, that wouldn’t be adultery. Waheed,” she says happily, “was very cross, because I gave George Clooney to Guy. He thought it was very unfair.”

Backbench Conservatives are allowed to be more forthright. I meet Nigel Lawson in the tea room. He sits completely still, emitting perfectly formed sentences amid the noise, like a man speaking directly to a page. “The power grab by the executive appeared before the hoo-ha over tax credits,” he says, “but it is true that the House of Commons has allowed the executive to grab far too much power.” The Commons, he adds, works much shorter hours, as a result of Blair’s “family-friendly” reforms. The Commons sits for only eight hours a day now, Monday to Thursday, and six hours on Friday. The Lords sits until after 10pm, every day except Thursdays, and sometimes all night. (He is careful to say that having more women in parliament is “great”.)

“The tendency of the government, increasingly, [is] to put major matters through secondary legislation, which should not be done,” Lawson says. “Secondary legislation is for the details.” This practice accelerated under Blair, and Cameron has gone further. It saves time, and it stops trouble – usually. “A lot of bills are not debated at all in the House of Commons. They fall to the House of Lords.” It is, he says, “a most extraordinary system that would not have been tolerated in my time at all.”

Is there anger towards the Commons? ‘Anger is putting it too strongly,’ Nigel Lawson says. ‘There may be a certain resentment.’ Photograph: Ben Quinton/The Guardian

Is there anger towards the Commons? “Anger is putting it too strongly,” he says, smiling very slightly. “There may be a certain resentment”.

Lawson despises the Liberal Democrats. “They are determined to wreck everything, because they are so bitter about the election. And anyhow, they don’t believe in an unelected chamber, so they wouldn’t mind [destroying its credibility]. The Labour party is much more responsible: not only are they not bitter and twisted in the way the Liberals are, they can envisage a time when they will be in government and they will want a system that works. The Liberal Democrats never expect to be in government again, so they couldn’t care less.

“The House of Lords is more useful than it used to be, because of the castration of the Commons,” he ponders. “There is a real risk that if you were to move away from this, you’d be in a worse position.”

But it won’t happen, Lawson thinks. “The Commons, naturally, is a very jealous place and it would never vote for a stronger second chamber,” he says. Or, as Patricia Hollis tells me, “Politics in Westminster is a zero-sum game. If we have more of it, the Commons has less of it. When MPs wake up to that, they are not too keen on seeing an elected House of Lords. Either it would collapse or continually challenge the other house. Stalemate.”

The Chamber during questions, with bishops wearing white. Photograph: Ben Quinton/The Guardian

The Lords is a place of paradox. It is, for instance, kinder to women than the Commons is. “There is no sense of boys throwing bread rolls and nanny will stop them. Women are heard here,” Hollis says. It is more inclusive. “If someone is disabled, and does not have a life in public speaking [but of public service], you can be heard.”

It is not an ideal place for a woman with young children who does not live in London, or for women who aren’t financially secure. Even so, both Labour and Conservative leaders are female, and there has never been a male Lord Speaker. One Tory peer tells me “there is male glass ceiling”, but he insists it is a joke and will not let me name him. It wasn’t always like this: when the 31st Countess of Mar arrived in 1975, she was not allowed to address the doorkeepers.

There are brilliant women here of the kind usually ignored by popular culture: shrewd, committed, post-menopausal. Some are elegant; some nondescript; some wear the rags of the dedicated female intellectual. They tend to specialise, gathering information from the charities and lobby groups they consider their constituents. Hollis tells me they “crawl over the legislation, literally line by line by line”.

I wonder if they seem authentic because they do not have to stand for election; if they did, would they all sound like Ed Miliband? I doubt it. Jenny Jones wants an elected chamber, “and nothing I have seen here has made me change my mind”. In her early days, she says, she was told off for running in the corridor and letting guests stand on the blue carpet in the Prince’s Chamber, which you can only “walk on, silently”. She wrote a blog complaining about the arcane rules: “Voting involves queueing in two lines in either the Content or Not Content corridors, saying your name to an officer to get your name ruled out in thick, black marker pen from the list, and then being counted off by a peer holding a drumstick. Musical, not chicken.” Afterwards, Black Rod came up in “his pantaloons and his fluffy tie” to tell her it was full of mistakes.

I hear another story, from the Conservative Jean Trumpington, who gave Tom King a V-sign in the chamber five years ago for alluding to her age. When she arrived in 1980, she was made a whip, but “hadn’t the first idea of what I was meant to be doing. Of course I made the most frightful hash of it at the dispatch box and was terribly embarrassed and terribly upset.” But the leader of the Labour party came up to her and said, “We girls must stick together. I’ve kept a file and I am going to go through it with you, and it will show you how you should behave and what you should say.”

These courtesies mask a forensic vigour that is terrifying and gratifying to watch. I see Sir Simon McDonald, the head of the diplomatic service, interviewed by the Lords committee on sexual violence in conflict. They bat questions at him on funding, priorities and processes; they are relentless. The diplomat taps his fingers with nerves and strokes his pen for warmth.

Looking down the royal staircase. Photograph: Ben Quinton/The Guardian

I accost Norman Tebbit in a corridor. He is ancient but skittish, giggling at his own jokes. It is difficult to hear him because William Hague is standing directly beside us, bellowing at two men. He ignores Tebbit, his voice charging up and down the corridor. His muscles bulge through his suit. He has the body of a dedicated superhero – Hansard Man – who ought to be suppressed.

Tebbit compares the Lords with the Commons, and says that here, “There is a vastly higher standard of debate.”

Hague shouts: “We’re really battling!” I wonder if he is talking about the government’s raging persecution complex, or judo.

“An elected House of Lords would have equal legitimacy with the Commons,” Tebbit continues, “and they would inevitably clash.” I ask if there are people who shouldn’t be here. “Yes.” He giggles silently. “Lots of them.”

“We’re really struggling!” Hague shouts, and goes away.

“I don’t think the prime minister understands the ethos [of this place],” Tebbit says. “We are not prisoners of our parties. We don’t obey blindly. There’s nothing I want that the whips can give me.” I believe him: he is 84, an unwhippable desperado. “At times I wonder whether Cameron is intent on making the Lords so ridiculous that you would virtually get rid of it.” Is he talking about the new peers? He grimaces. “Yes.” He won’t name names, but in the past 12 months, Cameron has appointed Michelle Mone, the “Bra Tsar”, and James Palumbo from the Ministry of Sound.

Since we are discussing ridicule, I ask Tebbit about John Sewel, the Labour chair of committees who was last summer photographed with a prostitute while wearing a leather jacket and an orange bra; he later resigned. “Everyone was very indignant about Lord Sewel,” Tebbit says, and giggles again. “But then he was merely being fashionable, wasn’t he?”

Lord Strathclyde says a fight between the Commons and the Lords has been brewing since 1999. Photograph: Ben Quinton/The Guardian

Just before Christmas, the Strathclyde Review is published. Strathclyde recommends the Commons is given the final say on secondary legislation. The Lords would be allowed to ask the Commons to “think again”, but would lose their power of veto. To me, it sounds barely stronger than the “regret motion” the Bishop of Portsmouth tabled over tax credits and which means: I feel bad, but what can I do? Strathclyde does not say how long the Commons would be given to think again: a year, a month, a day? Fifteen minutes?

On the day the Lords will debate his review, I have lunch with Lord Strathclyde in the peers’ dining room. The government has not yet responded to his review.

The walls are Liberal Democrat yellow, the carpets have an angry pattern, like a cruise ship. The food is adequate to good (they are not eating caviar off naked advisers). I see Charles Moore, Thatcher’s biographer, with a man who looks like a wizard; and Michael Grade, or Lord ITV. Sometimes the Lords resembles a collection of elderly celebrities aboard a ghost train.

Strathclyde’s enemies claim that his full name, Thomas Galloway Dunlop du Roy de Blicquy Galbraith, 2nd Baron Strathclyde, will not fit into a tweet, which isn’t actually true. His friends say he is very clever. At one point in Molly Dineen’s superb documentary, The Lords’ Tale, which details the expulsion of the hereditary peers in 1999, an elderly peer seeking advice collapses into Strathclyde’s office moaning, “Help me, Strathers... ” He reminds me very slightly of Boris Johnson: forensic eyes inside a pastiche of buffoonery. I tell Strathclyde that I have seen Dobbs. He hums with approval: “Dobbo is very good, very clear.”

Strathclyde says a fight between the Commons and the Lords has been brewing since 1999. It is merely a recalibration: the House – or “their Lordships”, as he calls his colleagues, fondly, rudely – is searching for the limits of its power.

The row over tax credits was “an emotional spasm, a reaction, a long time coming.” After 1999 there was a “new self-confidence, which I applaud”. (Others call it a “new professionalism”, after the men who liked to address other men only by nickname were cast out.)

We are really parliamentary worms. We take all the rubbish that comes from [the Commons] and it reappears more fragrant Michael Dobbs

In 1999, they were promised a two-stage reform: remove the hereditary peers, then get a fully democratic house. “We got the first but not the second; 17 years later, we are still waiting.” Strathclyde gives a slightly bitter laugh.

The last attempt at reform was in 2012. “The bill passed the House of Commons and it died a death. It suits the House of Commons to talk up the democratic mandate, but in practice they know it will give greater credence, and greater authority, to the second chamber, which is an attack on their privileges.”

Strathclyde spins through some of his personal preferences for reform: 80% elected, 20% appointed; large multi-member constituencies (based on the old Euro constituencies) for terms of 15 years, with five-yearly revolving elections. (Almost everyone has a plan for reform – the Countess of Mar once suggested an IQ test, “but how to do it eludes me”.) “But this is cloud cuckoo land,” he says. “None of this is going to happen any time soon.”

***

At 3pm, the debate begins. Strathclyde sits, defensively, in the second row, behind the government. During the debate, he exhibits an amazing range of facial expressions, depending on what the speaker is saying. Very occasionally, usually when a Liberal Democrat is speaking, he looks genuinely angry. The chamber isn’t particularly full, but it is January, and many peers are unwell; others are watching it on TV from their offices.

Eventually Strathclyde rises and begs for the House to practise “a self-denying ordinance”. If not, he fears “the House can virtually always defeat the government, and that way chaos lies and the patience of the House of Commons will be tried”. He uses his longevity in the House (he has been here since 1986) as a weapon.

And then, for six hours, over 56 speeches of six minutes each (they have to impose a time limit or they would end up back on camp beds), Strathclyde is pelted with fruit disguised as rhetoric. Not everyone opposes him, but the opposition is angrier (as he told me, the opposition runs this House). He smiles, laughs, waggles his head and stares, while the Conservative front bench emanate a sort of wracked reasonableness, which is their combined default expression.

Angela Smith for Labour accuses Strathclyde, courteously, of hypocrisy: as Tory leader of the opposition he rejected two statutory instruments. “So what has changed now that he sits on the government side of the Chamber?” The government’s case for weakening the Lords is feeble, she says: “An unnecessary solution to a fictitious problem.” This is a government “who fear opposition and loathe challenge. The noble Lord asks for responsible opposition. We provide that. What we seek is responsible government.”

Hollis echoes her: she points out that, between 2000 and 2010, the Conservative opposition, under Strathclyde, ran 11 fatal motions against the government, and two were successful. “No one had a tantrum. No one called for a review. No one proposed to legislate on the subject. No one threatened to create 100 peers.”

The Liberal Democrats join the charge, sort of – after first attacking themselves. Tom McNally calls the House “an affront to democracy. I may want to see this House reformed, but I have no wish to see it become Mr Cameron’s poodle, and a neutered poodle at that.” The House sighs.

And then the Tories join Labour in attacking the review. Patrick Cormack yearns for skeleton bills “to become a thing of the past: governments are not there to create Christmas trees on which ministers then hang balls”. Richard Balfe cannot see the point of a Lords that “cannot defeat the government”. (This is my problem with reform, too: the Lords costs £20m a year, which is a lot for a rubber stamp that loves you.) Michael Forsyth goes full Emile Zola: he thanks the tax credit rebels for rescuing the Tory party from itself, and says the response from the prime minister was “ungrateful. It was as if the captain of a ship which had been driven on to the rocks by the first mate, after being safely rescued, responded by inviting his crew to begin scuttling the lifeboat.”

Stowell rises. She has spent most of the debate with her legs resting on what looks like a tea tray on the floor. This does not look particularly odd in Pugin’s chamber – an elf playing a xylophone wouldn’t look odd here – but, even so, she has removed the tea tray. She is soothing but intransigent. She is a woman with two guns to her head.

But her answers are not good enough for the Lords. They begin to peck her. They say she is not listening, that no convention has been broken. Then – and this emerges like a tiny army wheeling out a cannon – they demand a joint committee. Labour peers jump up to interrupt her, again and again. Stowell slices her fingers sideways through the air, angrily: “I’m not, not, not, not, not... ” She doesn’t want to sit down, but she does, and they continue to harass her.

The public gallery is empty; even the agency reporter has gone home. I sit and watch them try to beat back the tide, and I feel a shocking gratitude.