On the top floor of the Ministry of Justice – an imposing brutalist structure just off St James’s Park, Westminster – lies one of the UK’s most powerful economic institutions. Since its creation by the former chancellor George Osborne in 2010, the Office for Budget Responsibility (OBR) has been charged with producing economic forecasts and forensically scrutinising government policy. In the polarised era of austerity and Brexit, its judgements have profound political implications.

The man who has chaired the OBR since its inception is Robert Chote. One recent morning I met Chote, 50 – tall, lean, dark-haired (with a marked resemblance to Serbian tennis star Novak Djokovic) – to discuss his work and the state of the British economy.

“It’s been great fun, there’s a good esprit de corps,” Chote said of his 30-strong team. Until the OBR’s establishment, the Treasury had sole responsibility for producing forecasts, which led to the justified suspicion that the forecasts were made to fit the policy, rather than the policy to fit the forecasts.

Chote spoke with gratitude of how the OBR had been accepted as a “permanent part” of the economic architecture. “Opposition parties tend to want us to do more, rather than less” (Labour has proposed that the body should audit party manifestos).

The politically independent OBR, Chote noted, is part of a “global phenomenon”. “I was in Cape Town in the summer and had an opportunity to meet my Zimbabwean counterpart. If ever I wake up in the morning and think it’s a tough job, that’s a salient reminder that it could be a lot more difficult.”

For eight years until 2010, Chote was director of the much-revered (and feared) Institute for Fiscal Studies. When Gordon Brown was presented by aides with the body’s painful post-Budget findings, he would exclaim with anger: “Chote!”

Before this, as economics editor of the Financial Times and a reporter at the Independent, Chote similarly tormented chancellors (a framed front page from 1992’s Black Wednesday – “Pound goes into freefall” – hangs above his OBR desk). It was while working for the International Monetary Fund in Washington, DC that Chote met his wife, Sharon White, a former Treasury permanent secretary, who is now chief executive of the media regulator Ofcom (the first woman and the first black person to hold the post).

Chote, the son of an Olympic javelin-thrower, studied economics at Cambridge University, where he was also chairman of the Social Democratic Party association (he stood as a county council candidate in 1989 following the Liberal Party merger). “I’m very lucky – I managed to get it all out of my system at the time,” he said of his past activism.

As a former journalist, Chote, unlike some technocrats, has an ear for an arresting phrase. The British economy, he told me, was “weak and stable, rather than strong and stable” (in reference to the Conservatives’ ill-fated 2017 election slogan). He warned: “There’s an evens chance of a recession in any five-year period if you look back at the historical experience. We’ve not abolished boom and bust.”

In its most recent set of forecasts (November 2017), the OBR dramatically revised down anticipated productivity growth. And though the latter has since increased at its fastest rate for six years, Chote cautioned: “These are noisy numbers, they go up and down, and we’ve had false dawns before”.

Chote, who lives in Corbyn’s Islington North constituency (in Tufnell Park), would not tell me how he voted in the EU referendum (“We have a secret ballot and I shall respect it”), he did not refrain from charting Brexit’s harmful economic effects: “In terms of the net effect on GDP, the hits to demand have outweighed the boosts”. (Following the referendum, the OBR forecast that the Leave vote would cost the UK £15.2bn, or nearly £300m a week, by 2020/21.)

Chote cited “most of the work that trade economists have done” as showing that the costs of leaving the single market and the customs union are greater than the benefits. “The reduction in openness likely with the EU is likely to outweigh any increase elsewhere.”

A permanent reduction in immigration, he warned, would increase the national debt. “Net inward migration tends to be a net positive for the public finances because inward migrants are generally more likely to be of working age than the native population.”

The OBR’s dull-sounding fiscal multipliers (which estimate the effects of tax and spending policies on growth) also show that austerity harms growth, while state investment aids it.

“Infrastructure spending has the largest direct effect,” Chote confirmed. “That’s basically because less of the spending leaks out into savings or imports.” He warned that growth would be “weaker” over the next few years as public spending cuts were still “intensifying”.

Even under the government’s fiscal rules, Chote noted, “there is room for spending more, there is some room for fiscal giveaway that would still be consistent with those targets”.

I ended by asking Chote the most important question of all: has he ever actually been mistaken for Djokovic? “No!” Chote insisted. “I think if you ever saw us physically next to each other there would be no doubt who was who, even if the headshots occasionally look slightly discombobulating.”