Just after midday on 25 November last year, Paul Johnson arrived at Millbank Studios, a pale stone building, used by news broadcasters, diagonally opposite the Palace of Westminster. Johnson, who is 49 and gangly, was riding a Brompton folding bicycle, his left suit trouser leg tucked into a red sock. (He claims to own socks of no other colour.)

Johnson is the director of the Institute for Fiscal Studies, an independent economic research organisation that occupies a unique position in British political life. Though other outfits attempt similar work, the IFS stands apart: when it comes to economic policy, its assessments have, for many, become the closest approximation to revealed truth.

“It is quite extraordinary in a way that it is regarded as the ultimate authority,” says Robert Peston, the former economics editor of the BBC and now the political editor of ITV news. “Basically, when the IFS has pronounced, there’s no other argument. It is the word of God.”

Johnson had been summoned to the broadcast studios on the occasion of the autumn statement, one of two announcements the Treasury is mandated to make to parliament annually. (The budget, which is typically delivered in the spring, outlines the government’s revenues and expenditure for the coming financial year; the autumn statement combines an update on progress since the last budget with economic growth forecasts.) Both events are major pieces of political theatre – and the IFS now plays a crucial role in their reception. If the chancellor is the impresario of the day’s performance, then the IFS has become Westminster’s most prominent – and feared – theatre critic.

For both the autumn statement and budget, the chancellor stands up in the House of Commons to deliver his speech at 12.30pm. Immediately after the speech, Johnson and his deputy commence a tour of the TV studios, offering immediate takes on the chancellor’s offering. Meanwhile, back at IFS headquarters in Fitzrovia, some of the brightest economic minds in Britain begin combing over documents that have just been released by the Treasury.

The 24 hours that follow are the busiest days of the year for the IFS. The institute’s staff must move quickly to calculate how much each new policy will cost, or benefit, certain sections of society, using a huge computerised model of the British tax and benefits system. More broadly, the IFS economists – many of whom are still in their 20s – assess whether the chancellor’s political rhetoric matches the harder reality of the numbers.

This work, which continues late into the night, fuelled by takeaway pizza, involves a search for where the Treasury has buried this year’s bodies – and an attempt to reconcile the complexities of academic economics with the media’s demand for straightforward answers. “Ranges are for cattle,” US president Lyndon Johnson supposedly once quipped to an economic adviser. “Give me one number.”

Basically, when the IFS has pronounced, there’s no other argument. It is the word of God Robert Peston

A select team drawn from among the IFS’s 40 full-time research staffers must race to build their findings into a presentation that is delivered at 1pm the following day – under the watchful eyes of all of Britain’s political, economic and media establishment.

What the IFS says about the chancellor’s budget determines the public narrative about what the government has actually done. At a time when the British media, public, and political establishment are all fiercely polarised, the institute’s findings are taken as gospel. No one can doubt its enormous authority. But how did one small economic research institute come to occupy such a commanding position in British public life?

Outside Millbank Studios last November Johnson, whose manner is cheerful and slightly awkward, met his deputy, a 41-year old LSE graduate named Carl Emmerson. Together they climbed the stone flagged stairs of the studios. At 12.30, when George Osborne stood to address the House of Commons, Johnson was ensconced in a booth in the BBC zone. In front of him lay a plate of sandwiches and a sheaf of freshly released documents from the Treasury. With headphones clamped over his ears, and his body hunched in concentration, Johnson watched the chancellor deliver his speech.

Osborne began by promising “far reaching changes to what the state does and how it does it”. The main thrust came 10 minutes later, when he abandoned a proposed cut to tax credits that had caused him considerable political vexation over the preceding months – partly thanks to the IFS’s initial criticism of Osborne’s plan.

In a summer budget on 8 July 2015, after the general election, Osborne had announced a cut to tax credits as part of a series of measures to shave £12bn from the welfare bill. The following day, at its post-budget presentation, the IFS claimed that the policy would cost 13 million families an average of £260 per year. A further three million would be more than £1,000 worse off annually. Johnson himself said the measure would hurt the poor “much more” than the rich.

The Financial Times, the Guardian, the BBC and the Independent all ran the story, prominently referencing the IFS. In October, the Sun launched a campaign under the headline “Tax credits cut bonkers”, using a single father of two named George Osborne – annual income £6,760 – to illustrate its theory. In November, the other Osborne did a U-turn.

When the chancellor’s speech was done, Johnson’s real work began. He sat for makeup and then toured the Millbank studios, first to a BBC set-up transmitting to BBC 2, the BBC News Channel and BBC Parliament. There Johnson sought to explain that Osborne’s change in policy was more cosmetic than the chancellor had suggested – that the overall cuts would in the long run be just as sharp as they were going to be when first outlined in July 2015. The key point was that, following reforms to the welfare system, new benefits claimants would receive less than at present. Johnson was playing his translator role, putting the numbers into – relatively speaking – plain English. “On that tax credit point, it is terribly important to be clear that he has changed nothing in the long run,” he said.

Later, Johnson and Emmerson headed to College Green, the rectangle of distressed grass where Westminster broadcasters stage their exterior shots. It was now past three o’clock and Johnson’s left trouser was still tucked into his sock. As he moved from live broadcast to live broadcast, he was constantly greeted by fellow members of the political and media establishment. On the Millbank Studio stairs he bantered with Robert Peston; on a lobby sofa outside a Sky News studio, Johnson discussed his choice of tie with Nigel Lawson, Margaret Thatcher’s longest-serving chancellor. But it was in the Sky studio itself that the IFS’s position as the ultimate explainer and umpire of British politics came into sharp focus.

The Sky anchor Dermot Murnaghan sat in a studio, framed by bars of illuminated red and blue. “Ah my God, the man that knows,” Murgnahan roared, off-air, when Johnson arrived. “I feel I’m here in a Penn and Teller show,” the anchor said. “And you’re going to explain what happens.”

IFS director Paul Johnson (right) and Nigel Lawson, a former chancellor of the exchequer, wait to appear on Sky TV. Photograph: Felix Clay/The Guardian

This week, following Osborne’s budget speech on Wednesday afternoon, Johnson and his researchers will work far into the night preparing their response.

The IFS occupies the top two floors of a 1960s office block on Ridgmount Street in Fitzrovia. The office is blandly corporate – staffers use PCs rather than Apple computers, and there is not a beanbag in sight. Venetian blinds hem the windows and bookshelves hold copies of Fiscal Studies, the institute’s house journal.

Most IFS staff – who invariably refer to their employer as “IFS” with the definite article elided – join straight out of undergraduate or masters programmes. Altogether 40 of the overall headcount of 60 are full-time “research economists”, while others are mostly senior academics with part-time affiliations. Their wardrobe is a contrast to the corporate look of the office – the younger staffers dress like members of a university debating team: Superdry T-shirts, Adidas trainers, zip-off trekking trousers. They are expected to scrub up for TV appearances, and a room in the basement, equipped with an ISDN line for static-free radio interviews, holds racked suit jackets alongside damp cycling clothes.

On budget days the institute obtains no special information before the chancellor’s speech, nor does it gain access to embargoed media briefings. Last November, by the time Johnson returned from Millbank in the middle of the afternoon, the work back at IFS headquarters was well under way.

The IFS’s autumn statement analysis was done by three teams, whose work corresponds to the areas they study in their day-to-day research. The most potentially explosive work took place downstairs. There, four young men clustered at desks strewn with papers, Fox’s biscuits, a copy of Accountancy magazine and a packet of Tyrkisk Peber, a liquorice candy popular in Scandinavia. Andrew Hood, James Browne, Stuart Adam and Rob Joyce work on “direct tax and welfare”, performing the distributional analysis that has become the IFS trademark, calculating how much policy change will cost certain groups.

Upstairs, two further teams were at work. The first, led by Gemma Tetlow, 34, focused on big-picture spending, while David Phillips, 30, led the final team, focusing on local government and devolution.

Johnson patrolled among the teams like a gallery owner, carefully polishing his treasures before unveiling them to the public. Johnson and Emmerson also placed calls to Treasury director Stuart Glassborow to seek clarification or check points of fact. One former Treasury official told me that there was always “an open line with a junior nerd at the IFS” on budget day, pointing out it is in the interest of neither side to get detail wrong.

At 6.30pm, the pizzas arrived, fresh from Olivelli, an establishment near the British Museum, and the staff gathered to eat in the common room on the third floor. Afterwards, as evening drew into night and deadlines approached, the atmosphere was calm. The scene resembled the performance of other complex tasks by an experienced team: shades of the aircraft cockpit or the operating theatre. (Exceptions are not unknown, however: Johnson told me a staffer once became so panic stricken on budget night they had to go home early.)

Before each team went to bed, the researchers sent on their draft slides for the following day’s presentation, on blue and green IFS-branded templates, to Johnson and Emmerson. The director and his deputy provided overnight feedback. Gemma Tetlow, the last economist standing, left the office at 1.15am.

Paul Johnson and his IFS colleagues eat pizza before analysing George Osborne’s 2015 autumn statement. Photograph: Felix Clay/The Guardian

The next morning, “IFS day”, as one former lobby correspondent terms it, started early. Johnson rose at six for the Today Programme, while staff were assembled in Ridgmount Street by about 8.30am. Those individuals selected to present – chosen by a process of relative democracy and a ruthless assessment of their social skills – wore suits. They rehearsed in front of Emmerson in the basement. The first slide of Tetlow’s presentation bore George Osborne’s face superimposed on Bob the Builder’s body. The caption read: “George the Builder: he can fix it (just not today – he’s enjoying the Sun)”. That morning, the Sun had triumphantly announced “Tax credit cuts scrapped by George Osborne in U-turn after Sun campaign.” In rehearsal Tetlow ran to twice her planned 20-minute length. “Serious butchering,” in her words, ensued.

At a few minutes before 1pm, the team decamped from Ridgmount Street and crossed Gower Street to the University of London campus. In the lobby outside the Beveridge Hall, much of the UK’s political and media establishment, identifiable by 270 plastic lapel badges, had gathered.

Johnson spoke first. “The first thing to say is that this is not the end of austerity,” he said. “This spending review is still one of the tightest in postwar history.” Tetlow followed, but this day belonged to Andrew Hood, who ran through his findings on how Osborne’s policies would affect welfare recipients. After the slides had slid, the press surrounded Hood, desperate to hear more from this young man who seemed to know more about the benefits system than anyone outside government. It was clearly exciting for Hood; he was relishing this moment after hours immersed in TAXBEN, the IFS’s computerised economic model.

As Johnson had done, Hood sought to explain just how the apparent U-turn by Osborne meant rather less than it seemed, especially over the long term. As with Johnson, though, not everyone got the message.

“Can I try something in journalese on you?” asked a reporter.

“Try,” Hood replied.

The reporter slipped into copy mode, and presented Hood with a news line about how many people would be affected by benefits changes in the next five years.

Hood mulled. “Let me think about this. Can I replace ‘Over five to 10 years’ with ‘In the long run?’”

“Umm, newsdesk says no,” the reporter answered, to collective laughter from his peers.

Hood continued to walk his recalcitrant audience through points of detail that they struggled to understand.

At one stage he broke from spending several minutes outlining exactly why a certain, apparently simple conclusion was in fact wrong, to examine the rolling TV coverage.

“Sky News have just reported exactly the line I’ve just told Sam he’s not allowed to print,” Hood lamented.

“Too late now,” said another reporter. “It’s news. Too late.”

The origins of the IFS go back to a bachelor weekend in 1967 at the Bell, a ritzy red-brick pub in the Buckinghamshire village of Aston Clinton. Four city men attended: tax consultant John Chown, investment manager Bob Buist, stockbroker Nils Taube and banker Will Hopper. Chown, who is now 86, recalls the Aston Clinton meeting as an opportunity “to have the four of us together for a weekend with no distractions apart from the fact that we were getting well fed and watered”.

The four men were bound together by a collective disgruntlement at the introduction of corporation tax by James Callaghan, then chancellor, in 1965. At the Bell, each man wrote an article on an aspect of tax, which they offered collectively to William Rees-Mogg, then editor of the Times. They expected the pieces to seep into print gradually, but Rees-Mogg ran them together on 10 April 1967, under the headline “A charter for the tax reform.” A reproduction of this piece now hangs in the basement of the IFS office.

Encouraged by the response to their article, in July the following year Chown, Buist, Taube and Hopper decided over supper at Stella Alpina, an Italian restaurant in Mayfair, to found an institute. The minutes Hopper made afterwards stated that the principal objective was “to study the economic impact of existing taxes and proposed changes in the fiscal system”. The intention was for an independent “research society”, rather than a “propaganda society”. Hopper proposed the name, including the use of “for” rather than “of”, a move that has persistently confused outsiders down the decades.

Johnson on his way to an interview about the autumn statement outside parliament. Photograph: Felix Clay/The Guardian

In 1969 the IFS was formally incorporated as a limited company and the following year, Dick Taverne, the Labour MP for Lincoln and former financial secretary to the Treasury, became the part-time director for the new body. In 1970s Britain though, the IFS struggled to make headway. As an organisation with no established track record it could not attract top-flight staff, and its work attracted little attention. In an attempt to raise its profile, in 1975 the IFS created the Meade committee, a wholesale examination of the British tax system led by James Meade, a renowned former Cambridge economist. Meade was assisted by two young economists: John Kay, who would go on to become director of the IFS, and Mervyn King, who would later become governor of the Bank of England.

The report it produced ran to 533 pages, took three years to prepare, and during this period Meade won the Nobel prize for economics for his earlier work on international trade and capital movement. On its publication in 1978, The Structure and Reform of Direct Taxation commanded five full pages of coverage in the Financial Times, and helped to solidify the IFS’s reputation. Yet funding was hand-to-mouth, relying mainly on corporate donations, with periodic infusions from grant-giving bodies such as the Gatsby Foundation. Evan Davis, now the presenter of Newsnight, arrived at the IFS in the summer of 1983 as an intern during his Oxford PPE degree, and returned full time the next year. He recalls on one occasion raiding a skip in order to acquire partitions for the office, which had relocated to Castle Lane in Victoria. (Though the partitions were not, as far as he remembers, actually used.)

In 1979, John Kay took over as director of the IFS. In his first few years in charge, it remained a scrappy outsider throwing stones at Whitehall. Relationships with government in general – and the civil service in particular, who perceived a trespass on their traditional expertise – were thorny. If some of the 1980s battles were part of the growing process of a new institution, Kay agrees that his personality may have also played a role in the IFS-government spats. “I am impatient to get things done,” he told me. After Kay and two colleagues, Andrew Dilnot and Nicholas Morris, published a book that was critical of the Inland Revenue, Laurence Airey, the chairman of the board of the Revenue, summoned Kay into an enormous ballroom in Somerset House, which then held the Revenue’s headquarters, and screamed at him for 15 minutes. When Kay interjected to ask for the specifics of Airey’s grievance Airey shouted back: “I’m too angry to give details.”

Kay, who is now a columnist for the Financial Times, recalls stepping down in 1986, emotionally exhausted by repeated tussles with the civil service machine. But under his tenure, the IFS’s public profile undoubtedly grew. In 1982 the IFS published its first “green budget”, which came out well before the budget, and aimed to lay out the issues at stake in the contemporary debate. The “green budget” continues to this day. The following year, the IFS presented its first “rapid response” to the budget, although at the first attempt the institute was not ready to go public with its “rapid” take until three weeks after the chancellor’s speech. Andrew Dilnot recalls limited media attention in the 1983 general election campaign. Four years later, the situation was very different. In the 1987 election campaign Dilnot found himself spending “most nights” at the BBC studios.

During the 1990s and early 2000s, the IFS’s public profile continued to grow, but the wrangles with government did not entirely vanish. When Gordon Brown was chancellor, he came to see the IFS, then under the leadership of Robert Chote, as a major antagonist. During that time, it fell to Damian McBride to present Brown with the “snaps” – immediate coverage of breaking news – of the IFS post-budget press conferences. Before McBride could speak, Brown would grab the paper from his hands and yell a single word: “Chote!” (Inevitably, “Chote!” became a catchphrase among Treasury staff.)

McBride believes the IFS was incentivised to pick holes in the budget because negative announcements drew more publicity. “It [the IFS] caused us more problems than the Tory party,” he told me. McBride was happy to fight back, however, once describing Paul Johnson to a journalist as a “failed Treasury economist”.

Alistair Darling, who succeeded Brown as Labour chancellor on Tony Blair’s departure in 2007, was less choleric, but he still acknowledged the potential problems posed by the IFS. “Of course it’s irritating and inconvenient when the IFS offers a critical view the day after a budget,” he said. “But frankly you have to take these things in the round.”

The IFS owes its prominence, in large part, to the media. It has been so successful at attracting press attention, in fact, that other thinktanks regularly contact Paul Johnson asking how they too might pick up such lavish coverage. But the secret of its success – aside from a well-drilled PR operation – is simple. First, many journalists are not confident with numbers; they want a reliable source they can turn to. (When I asked former IFS staffers for examples of the worst questions journalists had asked them, the wince-making responses included “How do you work out a percentage?”) The explosive calculation, last July, that Osborne’s proposed tax credit cut could cost some families upwards of £1,000 per year was made simply by taking the government’s projected saving and dividing it by the number of people who would be affected. That was primary school arithmetic – but it became potent only when it was stamped with the IFS imprimatur.

Second, the perception that the IFS is an impartial umpire means that harried broadcasters do not have to find someone to represent the “other side”. The official IFS position is that it has no ideological agenda – that its work is beholden only to data. (“You can referee data,” Andrew Dilnot likes to say.) The IFS has built its reputation on this rigid focus on microeconomics: they will assess the costs of new policies, or the impact of tax changes, but they studiously avoid “big picture” questions such as the causes of the 2008 financial crisis, or the wisdom of government borrowing. (Bonnie Brimstone, who heads communications for the IFS, routinely turns down press requests that fall outside this regular furrow.)

This narrow focus is the source of some criticism – particularly given the influence of the IFS on public debate about economics. Robert Peston argues that the IFS’s emphasis on microeconomic matters has steered British media coverage in a similar direction. “Everybody – BBC, ITV, newspapers – obsesses about what the IFS says about who the winners and losers are,” Peston told me. “Nobody really gets into the whole issue of whether austerity, for example, is good or bad for growth. Did George Osborne’s cuts at the start of the 2010 parliament make Britain richer or poorer?”

Paul Johnson (right) and Robert Peston (the BBC’s economics editor at the time) prepare to discuss George Osborne’s 2015 autumn statement. Photograph: Felix Clay/The Guardian

The Oxford macroeconomist Simon Wren-Lewis makes a similar point, arguing that the financial crisis, and the ensuing need for governments to undertake fiscal stimulus – borrowing and spending to spur economic growth – meant that one had to look beyond narrow microeconomic questions. After 2008, Wren-Lewis said, “you needed some macro knowledge to talk about the budget – about what fiscal changes would be effective at boosting demand – which the IFS does not have”.

Some left-leaning economists look with particular scepticism on the claim that the IFS has no ideology, arguing that the institute holds an excessive faith in the power of market forces. The tax expert Richard Murphy, a professor of political economy at City University who advised in Jeremy Corbyn’s campaign for the Labour leadership, said the IFS was “embedded in all the normal, standard pro-market assumptions that dominate conventional economic thinking in the UK and elsewhere”. Howard Reed, who worked at the IFS for nearly a decade and now runs an economics consultancy that has produced work for the Trades Union Congress and the thinktank Demos, took issue with the organisation’s view of itself as an impartial arbiter. “Suppose you’re a football referee,” Reed said, “and you don’t realise the pitch is sloped – you can try your best to be neutral, but one team might win five-nil just because of the way the pitch is set up.”

Allies of the current government have a different complaint with the IFS – that its distributional analyses, which show the impact of policy changes on various groups, only give a snapshot at the instant the changes are implemented, rather than their effects over time. This has led some on the right to argue that the IFS has not captured the way in which recent changes to welfare and tax credit policy may change behavioural incentives over the long term. “In a sense, those dynamic questions are probably more important,” said Rupert Harrison, a former IFS staffer who served as George Osborne’s chief of staff until 2015.

None of these criticisms are news to Johnson. “For nearly everything we do,” he said, “yes, we’ve got an economic framework around it – but actually we’re describing data. It’s the data that is doing the talking.”

In 1974, two American political scientists, Hugh Heclo and Aaron Wildavsky, published an influential book about the workings of the British state called The Private Government of Public Money, which argued that an extraordinarily small clique of officials, who know one another well, dominate Britain’s decisions about taxation and spending. More than 40 years later, while the IFS vaunts its independence, multiple ties bind Ridgmount Street, Whitehall and Westminster.

For instance, Paul Johnson served in the Treasury between his first stint at the IFS and returning as director. Prior to the IFS, Johnson was tutorial partner at Keble College Oxford to Ed Balls, the former shadow chancellor. Balls’s pre-government journalistic career at the FT moved in parallel to Robert Chote’s, who was then at the Independent. Chote preceded Johnson as director of the IFS. During his tenure at the IFS, Chote’s wife Sharon White was building a career at the Treasury that would culminate in her role as second permanent secretary at the department. In 1997 Gus O’Donnell, who would go on to become head of the civil service, was in Washington as the UK executive director on the boards of the IMF and World Bank; he held a wedding party at his house for Chote and White. Later, when Chote wrote to Gus O’Donnell to bemoan the Treasury briefing against the IFS, he ended his letter with the words “We have all known each other for a long time …”

In the three decades since Laurence Airey screamed at John Kay in Somerset House, the scrappy outsider has become an accepted part of the establishment – and a pathway for individual elevation to its ranks. Today the IFS recruits between one and four researchers each year. They receive 400 applications.

Nothing is a greater testimony to the growing power of the IFS than the decision made by George Osborne in 2010 to create a counterpart inside the government. On his accession as chancellor, he established an independent Office for Budget Responsibility (OBR), in order to take the business of economic forecasting out of the Treasury in the name of greater independence. Though the IFS does not itself engage in forecasting, and nor does the OBR do distributional analysis, by dint of its position as a quasi-external oversight body, the OBR did venture onto traditional IFS turf.

Paul Johnson in his office at the IFS in central London. Photograph: Felix Clay/The Guardian

Fraser Nelson, the editor of the Spectator, suggests that the motivation behind Osborne’s decision to create the OBR was to dent the public profile of the IFS, which many conservatives felt skewed leftwards, owing to the perceived tendency of British academic economists to favour big government and big spending.

Last year sunny economic forecasts by the OBR provided Osborne with an extra £27bn that permitted his U-turn on tax credits. In the days after the statement, some commentators suggested that the optimistic economic forecast was an example of possible OBR capitulation to political will. However, Robert Chote, the former IFS director who is now chairman of the OBR, is adamant about his organisation’s independence, and last month a Treasury select committee report indicated that while Treasury civil servants did try to lean on the OBR in the run up to the publication of the office’s earlier forecasts in December 2014, the OBR resisted.

Perhaps the best way to understand the relative positions – and prospects – of the IFS and the OBR is to see them as part of a decades-long struggle to tame an over-mighty Treasury. Britain is rare among European countries to have no separation between the finance ministry, responsible for short-term matters, and an economics ministry, which traditionally manages longer-term issues. As Robert Laslett, former chief economist for pensions at the Department for Work and Pensions remarks, attempted governance mechanisms that do not have an absolute separation from mother Treasury tend to run into trouble. They are, in his words, “piglets too close to the sow”.

The IFS today occupies a quasi-constitutional role in British life, but without the scrutiny on management and funding that applies to formal government bodies. Its separation from government may be one of the best explanations for its success.

For the IFS itself though, life goes on. On Monday, four months after the autumn statement and two days prior to the budget, Johnson and his team were gearing up to do it all over again. With rumbles afoot in the stock market, and following the Bank of England’s decrease to its earnings and peak growth forecasts, he suspected the OBR would downgrade its economic forecasts. “To the extent that things are changing in the economy it looks like they’re changing in the wrong direction,” he said. However, Johnson emphasised that ultimately he and his team would not know how things would go down until Wednesday, when George Osborne stands up in parliament. “I don’t have any more idea than anyone else I’m afraid,” he added. “So we’re sort of as ready as we ever are.”

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.