At the top of the escalators in the concrete caverns of Westminster underground station, most passengers either make for the exit straight ahead on to Westminster Bridge or duck right to come out on Whitehall. Just a few, though, will head left before taking a sharp right along the underground walkway and disappear through a badly lit revolving door, on the inside of which is a police guard. This is one of the lesser-known ways into the bubble that contains the British parliament.

The insider’s guide to Westminster: from Portcullis House to the Burma Road Read more

Having passed through the security gates – no need for any of the hassle of bag-searches or x-ray machines that visitors have to endure at the main entrance – you come out into another underground walkway. Turn left and you walk through a tiled passageway between a stone unicorn and a crowned lion, to a colonnade that takes you towards the main House of Commons debating chamber. To your right is New Palace Yard and the MPs’ underground carpark where Conservative MP Airey Neave was murdered by the Irish National Liberation Army in 1979; to the left is Speaker’s Green and Speaker’s Court. On most days, this walkway is crammed with people walking purposefully in both directions. No one inside the Palace of Westminster would be seen dead doing anything at all less than purposefully. The most purposeful, and therefore the ones who can afford to walk at a slightly more languid pace than everyone else, are the MPs. They are recognisable by their green-and-white-striped security passes, if not always by their faces; some MPs seem to find coming to Westminster a bit of a drain on their time and prefer to stay away.

At the start of the new parliament next Monday, there will be an additional sub-group of MPs: the new intake. In addition to the passes, they will be wearing little green pin badges - think Paddington’s “Please look after this bear” note from his Aunt Lucy. Ideally, they will have all been given a brief induction tour by a member of the permanent Westminster staff, which will have been forgotten before it’s even over, and the lucky ones will have been assigned senior MPs from their own party as mentors. They will still look bewildered and lost, because they are. “It takes about a year for them to find their way round,” one of the many machine-gun-carrying policemen confided. “You can tell, because that’s when you become invisible to them. At the start, they are always friendly, saying hello and holding doors open. Then it stops. I’m sure they are not bad people, but they just become part of the bubble.”

So who are the members of this new intake? Increasingly, they are career politicians – men and women who have eyed up a career in politics since they were at university and have done everything possible to fulfil that dream. Paper rounds, bar work? Um no. They’ve worked as unpaid interns, schmoozed the right people at party headquarters and even – though not ideally – paid their dues in local politics. There are always exceptions, of course. Rotherham MP Sarah Champion, who was first elected in 2012 at the age of 42, had run an arts centre and a children’s hospice before running for office. But MPs like her are becoming harder to find. For all of them, though, no matter how entitled they already feel, Westminster is a working environment like no other. What follows is a rough navigational guide.

Of the population who look as though they know, more or less, where they are going, you can tell the peers by their red and striped passes. They tend to have a distracted air, as if focusing intently on whether the brackets in the International Development (Official Development Assistance Target) Bill that they will be debating later are in the right place. The media are identifiable by their brown badges, the Palace staff by their grey ones. Lords’ assistants and researchers are issued with straight reds, but the most terrifying species of all are those with straight green. These are the MPs’ assistants and researchers, known as spads, almost all of whom are under 30, look as if they are under 20 and, if all goes well, will be MPs themselves within five years. They walk at twice the pace of everyone else and their eyes burn twice as fiercely; they have the certainty of their convictions yet none of the responsibility for the consequences. They also might that morning have drafted a clause in a bill that could make life either a bit better or a bit worse for hundreds of thousands of people. It doesn’t really matter too much which. What counts is that they’ve done it.

The MPs’ assistants and researchers walk at twice the pace of everyone else and their eyes burn twice as fiercely

After three minutes in which the chamber is closed for prayers, each day’s session opens with the Speaker announcing departmental questions. The Speaker is an MP who has formally renounced his party affiliations to preside over the parliamentary procedure and maintain decorum in the House with cries of “Order! Order!” issued in a voice between strident and frankly bored, and is elected – or re-elected – on the first day of every new parliament. Having mounted a graceless, failed coup to remove the current Speaker, John Bercow, on the final day of the last parliament, the Tories have privately agreed among themselves not to reoppose his re-election next Monday.

During busy sessions, the whips, whose job it is to maintain party discipline and make sure MPs vote as expected (party leaders do not like surprises), maintain a constant vigil on their own side’s MPs. Even in the dullest, most poorly attended sessions there will always be at least one whip on view. In the chamber, they often stand behind the Speaker’s chair, clipboard in hand, checking the roster and bestowing a wintry smile on a favoured few; outside the chamber they are rather more hands-on. Order is maintained through any means necessary: sometimes a phone call or a private visit to an MP’s office; an appeal to decency, or to ambition; if all else fails then blackmail will do. The whips are a breed apart: apparatchiks who can be relied on for their loyalty but not necessarily sharp-witted – or telegenic enough – for ministerial office. Only about half of whips ever get another job in government (Michael Gove, the former chief whip, is one honourable exception).

Departmental questions offer the opportunity for a cabinet minister and his or her junior ministers to be held to account for what their department is doing. Although the legal requirement that MPs will not tell lies in the chamber does not equate to them necessarily telling the truth, some specialist knowledge is required. A transport minister needs to be a mixture of the Fat Controller and a GB satnav as well as a skilful politician. A granular familiarity with the road and rail networks is essential to head off awkward questions about traffic improvement on the B668 or delays to train services on the Great Western main line. One Lib Dem MP inquired many times over the years about the possibility of reopening Corsham station in his constituency. As it has been closed since 1965, the answer was none whatsoever, but it never hurt to ask. That MP lost his seat at this election: not much breath is being held to see if his Conservative successor continues the tradition.

Westminster is frequently described as a bubble. It’s less fragile than that. Bubbles are more easily pierced. Rather, it is a hermetically sealed environment that encourages introspection and self-importance, and from which the outside world is largely excluded. It is the centre of its own universe. Politicians frequently talk about how much they enjoy meeting the public, but they seldom look more awkward than when meeting real people, and their minders go to great lengths to ensure they don’t have to.

Not that all politicians are as self-interested as many outsiders believe. Venality and entitlement merely creep up on them. When politicians say they entered public life to contribute to society, they are telling the truth – if only in part. Few people embark on careers with the express intention of making the world a worse place. But becoming an MP requires a certain, well above average, level of ambition and self-belief, both of which get amply fed and nurtured by Westminster itself. It’s hard not to feel important when there are liveried men and women at your bidding, armed officers on patrol to guard you and your every word is listened to and sometimes even admired.

Westminster goes out of its way to feed these ambitions. The higher up the food chain a politician can get, the more rarefied and detached the atmosphere becomes. Departmental limousines are on hand to drive you a few hundred yards. Traffic lights are always on green and you never have to look for a parking space again. Just as the Queen must believe that every toilet in the world is kept in pristine condition with a new lavatory seat installed every day, so too politicians can easily lose touch with reality. They are busy, busy people, who forget that one small compromise often leads to another. They never intended to lose contact with ordinary people, it’s just something that happened through a process of accretion. And power is very seductive. Why become an MP just to be a backbencher when there are so many more possibilities higher up the ladder? Junior minister, cabinet minister … But to get there you have to play the game. And the game has strict rules that must be learned.

Facebook Twitter Pinterest Illustration by Daniel Haskett

Most MPs avail themselves of the chance for a short nap during Business questions, after which the leader of the house – a cabinet post generally given to senior politicians in the twilight of their careers – and the shadow leader gently, and good-naturedly, make fun of each other. This tradition applies more towards the end of a parliament when a government has run out of things to do and is trying to find ways of filling up the parliamentary calendar without appearing to be wasting everyone’s time. At the beginning of the parliament, when a government is trying to implement as much tricky legislation as possible while it still has the public’s goodwill, the exchanges are more pugnacious. The nature of this session is also defined by its two participants. William Hague and Angela Eagle were quick‑witted adversaries at the end of the last parliament. On past form, the new Conservative leader of the house, Christopher Grayling, may be rather more dour.

Various press secretaries ensure that there is never nothing going on in Westminster. The news agenda continues around the clock; a misplaced sentence on the Today programme can be headline news within minutes. The news, itself, comes in several layers. The official government business of the day is passed on through daily lobby briefings, usually given by the prime minister’s spokesman. Matters are kept entirely factual and to the point: who is doing what and when. The lobby briefings take place in a small room near the top of the House of Commons, attended by hard political news reporters rather than the commentariat. Below that – both physically and metaphorically – everything gets a little murkier.

The press occupy a third-floor corridor, known as the Burma Road. How it came to be called this has become lost in memory, though the Guardian’s longtime political editor, Michael White, suspects it may have been a piece of second world war codology, coined by veteran press gallery secretary, John Deans. At the very least, it’s a symbol of suffering. To reach it, you either have to walk up a rather bleak, stale‑smelling, winding staircase or take a small, wood-panelled lift, that isn’t quite as old and decrepit as it looks. One hopes. The luckier organisations have offices with windows that look over New Palace Yard; the Guardian’s quarters face the opposite way and, until comparatively recently, were windowless. Politicians seldom visit the Burma Road in person. But their spin doctors, policy advisers and press officers are an almost daily distraction, attempting to portray events to their leaders’ advantage.

It’s a dance that is generally played out with good humour and politeness; many party press officers are former hacks themselves, hired because they understand the system and know the characters involved. Tony Blair appointed former Daily Mirror journalist Alastair Campbell as his director of communications; David Cameron made former News of the World editor Andy Coulson his communications supremo. But not every appointment is quite so divisive as these. Cameron’s current press secretary, Graeme Wilson, the former deputy political editor of the Sun, is one of the most likable men in the business – wry, funny and very bright. Unlike outside Westminster, everyone really is in it together. There’s no point falling out with people you’ve got to work alongside every day, unless it’s really worth it.

There was a time at the beginning of the last coalition when the Lib Dem spinners had a certain swagger and authority. That moment has now rather passed: a story about Danny Alexander’s plans for the economy has lost much of its value now that he is no longer in Westminster and the Lib Dems hold just eight seats. They will still get a hearing, but their envoys are likely to be much more deferential and grateful for any interest. The Conservative spinners are, without fail, polite; it’s as if finishing school was part of their training. For the most part, Labour’s were too, though Tom Baldwin, Ed Miliband’s right-hand man, was known to be a little more animated at times. The louder he got, the less attention people paid.

The press occupy a third-floor corridor – the Burma Road – How it came to be called this has become lost in memory

If the news is not good, the spinners’ task is to try to minimise the damage by telling anyone who will listen that down is up and not everything is quite as bad as it seems. In 2001, Jo Moore, a spin doctor in the Department of Transport, was forced to resign after sending out an email on 9/11 that said: “It’s now a very good day to get out anything we want to bury.” Her mistake was to put it in writing. Trying to bury bad news is still standard practice. If a government has good news, it will usually try to make sure that dominates the headlines for the day, so other good news may be held back. Having two pieces of good news on the same day is a waste of good news. Bad news, however, is best dealt with wholesale. If the government releases several reports on the same day, it is odds-on that there is something it doesn’t like in at least two of them.

Monday was one of the biggest political news days of the year, with the prime minister announcing his cabinet and addressing the influential 1922 committee, the Labour party beginning the very public postmortem into its electoral defeat and the 56 Scottish National party MPs arriving at Westminster for the first time. So it was the perfect day for Ukip to slip out the embarrassing announcement that Nigel Farage would not be resigning as party leader after all, after promising to do so the Friday before. It was tantamount to an admission that Ukip really was a one-man show and that without Farage the party could all but disappear. As one person wrote on Twitter: “Leaving on a Friday and coming back to work on a Monday is normally called a weekend.”

Debates in the Commons are often more leisurely affairs than the cut and thrust of both departmental and prime ministerial questions, and the exchanges are invariably better informed – apart from those occasions when an MP is trying talk down a bill by running down the clock; then the digressions can be painfully dull. All parties are prepared to use these procedural ploys when it suits them so the disruptions are generally taken with cordial acceptance; they are just one of the many ways the system can be gamed. If a vote is required at the end of a session a division bell rings on the TV screens, on alarm systems throughout the parliamentary estate and in selected clubs, pubs and restaurants in Westminster. MPs have eight minutes to get back to the lobby to vote before the doors are closed. During this time the party whips dash around with their clipboards: they like to know where all their MPs are at any given time and don’t like anything unexpected in the voting lobbies.

Many MPs may frequently have little idea of the nuances of what they are voting on and are unlikely to have been following the debate closely. In these instances, they aren’t being paid to think. They are being paid to vote the way their party wants them to. Then there are the so-called “Big Beasts” of parliament: politicians such as Ken Clarke, Michael Heseltine, Gordon Brown and John Prescott. Ed Balls and William Hague also fitted the description. Big beastdom can be awarded for success in any number of categories, such as longevity in parliament, intelligence, humour, personality and quality of public speaking. Boris Johnson, who has come back to parliament this election after taking time out to be mayor of London, is one such big beast. He is a man with charisma to whom you want to listen – a man who can be guaranteed to liven up proceedings, even when he is talking bollocks.

Big beastdom should not be a hard-to-acquire honorific, it’s just that so many of the men and women in parliament are so unremarkably grey. They have channelled their ambition into saying yes and no at precisely the right moment and staying firmly on message. They have got where they are by being boring. Sometimes there is genuine pathos in this. Generally speaking, there are some ministerial posts that MPs can’t wait to leave. Pensions minister is one such post. Yet in the last government, Lib Dem Steve Webb more than just understood pensions, he lived and breathed them. He thought pensions were the most interesting subject in world history. Webb lost his seat at the election. Parliament won’t miss his greyness, but it will miss his expertise.

There are groups of MPs who – while never achieving big beastdom – are still less willing to play by the rules. These are known as the awkward squad. In the Labour party, they are often former trade unionists, such as Dennis Skinner, who still resist any move away from the party’s core principles of the 1970s. In the Tory party they tend to be hardcore Euro-sceptics, some of whom have flirted with the idea of joining Ukip. With the Lib Dems now occupying so few seats, anyone in that party can be as awkward as they like. The difficulty for them now will be to get anyone to notice.

These squads are, by and large, resistant to any threats their party bosses may make. They also tend to grow in number over the course of a parliament, as disappointment and disillusionment set in. A few, especially on the Conservative side, can be bought off with promises of a knighthood or a peerage, but most derive their main job satisfaction from being independent-minded. They may have already slipped out of favour and been demoted from a ministerial post. Perhaps they never did aspire to high office, or even much of an office, at all. While some of the MPs’ offices on the estate are relatively airy, some are little more than underground bunkers. Douglas Carswell found he had been allocated just such a dungeon on his return as a Ukip MP after defecting from the Conservatives (he had rather a nice office before his act of treachery). The awkward squad also often have quite safe majorities, are popular with their own constituency parliamentary party and so are in many ways untouchable.

The cafes in Portcullis House are often full of MPs who are meeting campaigners, lobbyists, people in difficulties – even school parties – from their own constituencies. The modern MP has to be part therapist, part tour guide. These may not be the most exciting aspects of the job, but they are essential. And nothing an MP ever does entirely goes to waste. MPs like to observe and be observed. The Westminster hierarchies run deep: to get a nod from a cabinet minister can make a backbencher’s day, especially if there’s a crowd of colleagues present. The open spaces of Westminster are for display and outward shows of charm. The backstabbing and plotting takes place far away in corners of late-night bars and by text and email. Electronic communications have superseded the old system whereby journalists used to wait around in the central lobby hoping that politicians would walk past to whisper poison in their ears.

The backstabbing and plotting takes place far away in the corners of late-night bars and by text and email

Perhaps the best way an MP can get noticed is through being asked to join one of the House of Commons select committees that take place either in one of Pugin’s gothic committee rooms in the palace or one of the modern wood and leather rooms in the Commons extension, Portcullis House. There are about 30 such committees tasked with examining government policy or proposed new legislation. Anyone from former prime ministers, serving ministers, bank officials to the families of the 15-year-old girls who ran away to join Isis in Syria can be asked to give evidence. Even the mayor of Calais was summoned last year.

The home affairs committee is very much the Keith Vaz show. Vaz is a curious political animal: an MP who is both incredibly vain – he will do anything for a headline – and yet one of the great survivors. In 2002, he was suspended from the Commons for a month after an inquiry by the standards and privileges committee found he had made false allegations against a former policewoman. And yet he has now reinvented himself as one of the guardians of Westminster’s moral probity.

Vaz dictates the mood for each session – sometimes inquisitorial, sometimes deeply concerned (bizarrely Vaz is often at his most scary when he’s trying to show how much he cares) – and he makes it clear that his view of the reliability of a witness’s evidence is what counts. He sometimes seems to regard the interventions of his colleagues as a distraction. To be fair, sometimes they are distracted. I spotted Lib Dem Julian Huppert Googling himself on his iPad in one session: he is by no means the only MP to have doubted his or her own existence (after the last election, he has indeed vanished).

Illustration by Daniel Haskett

Margaret Hodge also imposes her personality on the proceedings of the public accounts committee (PAC). She, too, is a masterpiece of her own reinvention. She has apologised for the “shameful naivety” of Islington Council in the 1980s, when she was leader, in its dealings with child victims of sexual abuse. The Times has also recently raised questions about her offshore tax arrangements. And yet she is pure box office (she described Google’s tax avoidance as “evil”) and must be close to big beastdom. When Rona Fairhead, the former HSBC director, was interviewed by the culture committee prior to her appointment as chair of the BBC Trust, one of the main concerns the committee had was that she might find the workload a bit tiring. Hodge said Fairhead, who had only been approved as chair of the BBC Trust by the DCMS select committee several months earlier, was unfit to hold that office and should reconsider her position.

Not every committee can always provide the theatre of the HSBC inquisition – some, it has to be said, are terribly dull. But their proceedings are genuinely democratic: the difficult questions do generally get asked as the lesser-known MPs temporarily lose their two-dimensional constraints and display unsuspected depths of intelligence, humour and conscience. Accountability and transparency are not dead inside parliament. They just sometimes take a little finding.

On a particularly rainy day, it’s not uncommon to find strategically placed buckets throughout Westminster – not even the grandest of grand corridors, halls and staircases are immune from the indignity. Much of the grandeur is barely even skin-deep; the roof and ceilings leak, the masonry is crumbling and not even the small army of builders, conservators and contractors who are permanently on site can hold the building together. In political parlance, Westminster is not fit for purpose. What to do about it is another matter. To restore the palace to full working order would cost about £3bn and require the Lords and Commons to relocate for several years. This is not the kind of decision any government wants to make. Parliament spending money on itself never looks good at the best of times. And where does it move to? Birmingham? Manchester? Liverpool? Whatever decision is made is going to upset someone. Far better, then, to do nothing. The only thing that might rush a government into action was if a piece of the Commons roof fell down during prime minister’s questions and took out several backbenchers. Though it might depend on who they were.

Even with the new female intake, Westminster is still unquestionably male. It looks as if it was designed by and for men of a particular privileged class, its halls and chambers resembling a cross between a public school, an Oxbridge college and a gentlemen’s club to which only the already entitled are welcome. When Sir Malcolm Rifkind made his first appearance in the Commons after being filmed touting for outside work earlier this year, the Tory backbenchers gave him a huge cheer before he spoke. Decent chaps have to stick together and all that.

For women, Westminster can be an alien world. Sexism is so everyday and institutionalised, most MPs fail to even realise its existence. Even now, some MPs cup their hands across their chest and jig them up and down – off-camera, of course – when a woman gets up to speak. It’s like a reflexive, unconscious herd-like response from people who wouldn’t dream of behaving in such a way anywhere else. Even the prime minister sometimes can’t help himself. In one heated exchange during PMQs in 2011, he told Angela Eagle, then shadow chief secretary to the Treasury, to “Calm down, dear.”

For women, Westminster can be an alien world. Sexism is so institutionalised, most MPs fail to realise its existence

The procedures of the Commons are sometimes arcane. Last year, for instance, three MPs hatched a plan to sleep by turns in the room adjoining the office of the Commons clerk responsible for allotting the last remaining slots for private members’ bills, so at least one of them would be first in the queue when he arrived. They are, however, just about functional once you know how the system works. Sometimes, though, this functionality gets lost in smoke and mirrors. Never more so than at prime minister’s questions, the theatrical centrepiece of the Commons week in which the prime minister is meant to be held to account for the government’s actions, yet seldom is. Somehow, it manages simultaneously to be both democracy in action and democracy inaction.

Before PMQs, which take place at midday on a Wednesday, there is a brief round of departmental questions in which the Scotland, Northern Ireland and Wales offices get their occasional half-hour in the sun. This is the session in which Westminster proves it really doesn’t care about the regions at all. As the clock ticks round to midday, the chamber begins to fill up with MPs from all sides; none of whom feel obliged to discontinue a conversation they were having when they walked in. Come 11.55am, the chamber is drowned out with 300 different conversations while the unfortunate minister carries on talking inaudibly.

At midday the Speaker brings this particular form of torture to a halt and announces: “Questions for the prime minister.” After a couple of warm-up questions from backbenchers to quieten the crowd, the leader of the opposition is called to the dispatch box. The noise levels on both sides of the house increase significantly. The questions vary week on week but the responses do not. The prime minister is not obliged to answer, merely to say something. Traditionally, Cameron either answers a different question to the one he has been asked or asks why he hasn’t been asked a question about something he has done well. When he was leader of the opposition, Miliband got to ask six questions. The new Labour leader will only get four; the other two will go to the Scottish National party, which is now the Commons’ third-largest party. According to how the protagonists are doing, their supporters either jeer, yell or make farmyard noises.

Once the exchanges are over, MPs and the political media start tweeting frantically, with a variety of scorelines. Labour MPs always reckon their man has won, while the Tories always say that theirs has. To see out time, Cameron has another 20 minutes of questions from backbenchers. The Tories tend to ask him if he happens to be aware how well his long-term economic plan is working in their constituencies. He does. Labour MPs ask him if he knows that his long-term economic plan isn’t working in their constituencies. He says they are misinformed.

Shortly after 12.30pm, the Speaker puts everyone out of their misery. The crowded chamber and press gallery empty. In the ante-room the press lobby forms a huddle around the prime minister’s press officer who tries to explain what the prime minister had really meant and what he might have meant had he been asked something else entirely. The game goes on. For everyone working inside it, Westminster is not merely the seat of government, it’s also a test of endurance and nerve. Daily survival is often all that matters. Tomorrow can look after itself.

• Follow the Guardian Long Read on Twitter: @gdnlongread