A month into 2016, two men were dining one evening in Rotorino, an Italian restaurant in Kingsland Road, Hackney. It rapidly became clear that one of those at the table was deeply distracted, taking regular calls and going outside into darkness for 10 minutes at a time to remonstrate. Each time he returned, he was more contemptuous. An operation was under way to remove him from his job. He started blaming dysfunctional idiots. The name Bernard Jenkin came up a lot. As the punctuated evening wore on, and his linguine rapidly cooled, the dishevelled figure popping up and down from the table gradually relaxed, telling his patient companion he thought it was going to be OK. “The donors are going to see them off.”

It was probably the most concerted effort of many that have been made to marginalise Dominic Cummings from British politics, on this occasion his leadership of the Vote Leave campaign in particular. If it had succeeded, it is quite possible the leave campaign, instead offering a one-dimensional message about immigration, would have lost. But history turned a different page.

Now Boris Johnson has decided to appoint Cummings to Downing Street, a brilliantly kept secret that his new adviser will have adored. It propels possibly one of the most controversial and fascinating figures in British politics right into the heart of government. He has a chance to finish what he started back in 2016. The era of lobbing bricks from the sidelines is over.

The unique aspect of Cummings is that he is so many different things at once. He has a fascination with strategic long-term challenges akin to that of Oliver Letwin. He has base campaigning instincts to rival Steve Bannon, and a similar contempt for Whitehall systems failure to Louise Casey, Tony Blair’s antisocial behaviour tsar. But in one respect he is completely unmatched: his ability to conduct feuds, or to put it another way, put into practice his highly developed theory about achieving change in unresponsive bureaucracies.

For Cummings, corners exist only to be cut and orthodoxies to be challenged.

David Cameron, bewildered by the endless incoming fire from the education department during Cummings’s period as a special adviser under Michael Gove, famously described him as a “career psychopath”. Bizarrely the two never met.

Equally Nick Clegg, a figure of risible contempt to Cummings, found himself under a ceaseless barrage over a proposal on free school meals. “Dreamed up on the back of a cigarette packet” was his most friendly description of the policy. Clegg said Cummings had “anger management problems”. Even the chief inspector of schools Sir Michael Wilshaw, once seen as a scourge of teaching mediocrity, was derided by Cummings as out of his depth and insufficiently radical.

Gove and Cummings in 2012. Photograph: Oli Scarff/Getty Images

Gove, polite to a fault, would often feign ignorance of his adviser’s methods, but knew full well the dark arts that Cummings deployed to get his master’s way. It was only after Cummings – no longer in Gove’s employ – in one of his highest-octane outbursts called Cameron “a sphinx without a riddle”, his chief of staff Ed Llewellyn a sycophant and his director of communications Craig Oliver clueless, that Gove felt obliged to distance himself. The caustic character assassination was Cummings at his best.

His voluminous writing, if it is not bogged down in eugenics, mathematics or causation theory, specialises in a fin de siècle absurdity. “As the black flags of Isis fly and Putin seeks to break Nato, William Hague poses for the cameras with Angelina [Jolie] and Cameron’s closest two advisers stick with the only thing they know – a 10-day planning horizon (at best) of feeding the lobby (badly) and changing tack to fit the babbling commentariat (while blaming juniors for their own failings).”

Now, as he knows full well, there is no one left to blame, but himself. He will act as the interface between Gove and Boris Johnson.

Despite predictions that Gove would be put in charge of a major government department, such as Communities, he has instead been put into the Cabinet Office, the government engine room. The Cummings-Gove diarchy is back in harness, preparing for no-deal Brexit, and possibly an early election. The team that brought free schools and expansion of Blairite academies in the face of what they dismissed as “the blob” are now going to get ready to cut the umbilical cord with the single market.

The ministerial civil service code will not quite be chucked in the bin, but it is not likely to be the most heavily thumbed manual in the months ahead.

It is not just that Cummings hates individual civil servants – Olly Robbins and the deceased Lord Heywood were particular objects of his ire – he hates the whole Northcote-Trevelyan civil service system and believes it is one of Britain’s decrepit institutions that will most benefit from Brexit. “It keeps out great people, it hoards power to a small number of people who are increasingly crap. And the management of the whole thing is increasingly farcical, like that of any closed bureaucracy keeping its perks. It cannot manage public services, it cannot deal with counter-terrorism. It’s programmed to fail – and it does.” Brexit, he told the journalist Tim Shipman, “would force people to think about these things rather than being in the brain dead stupor they have been in the last 20 years where Whitehall just thinks about being in a Brussels meeting”.

The temptation will be to see Cummings as the next Steve Hilton, the modernising free-thinking guru to Cameron who arrived in Downing Street in 2015, only to leave spent and defeated after two years. The two men after all share the same betes noires: distant, unaccountable power, thick-witted rules, many from Europe, gold plated by slovenly bureaucrats. Cummings views Europe as like the Ming dynasty: it dwindled because it cut itself off from outside influence.

The problem in Hilton’s case was, in the words of Giles Wilkes, then a Liberal Democrat adviser, that he developed “policies doomed to dissolve on contact with the world they hoped to change”. He became a “fizzing catherine wheel of chaotic political force spiralling through Whitehall. Prejudices formed in a blink threatened entities that had stood firm for years.” It emerged that Hilton knew more about marketing than about the free market.

But Cummings is very different to Hilton, much more directed, focused and target-driven. He has also been around the Conservative party scene much longer. He will have written a plan, not a vision, even if the timing was not perfect, since he had to delay a minor operation to be ready to go into No 10.

So what drove him to return? One friend says he simply did not go into government after the Vote Leave triumph because his wife, Mary Wakefield, had just had their baby (during the referendum) and she pretty much forbade him.

He could often be seen in the morning outside his local cafe with his baby. Occasionally, if it was hot or indeed very cold, he sat with a scarf over his head, bent double over a book. Anna Karenina, maths and Bismarck are his three obsessions.

Wakefield remains a very close friend of Johnson from when he edited the Spectator, but Cummings was not in the loop when Gove decided first time round to have a tilt at the leadership, an event that soured the Johnson-Gove relationship for nearly a year. His child was a few months old, and his energies elsewhere. Now he thinks the risk of a Corbyn government is high if the Tory party opts for a soft Brexit.

So will he survive, given the enemies he has made? Cummings’s caustic pen, and tongue, can also hide his true character. “He is also the first person to say if he doesn’t know something. He has a gentle manner. Truly modest. A good listener. He is very keen to find brilliance in unusual places,” says one friend. “He is neither hierarchical nor deferential. But alongside that, he makes no apology for brutal political judgments. He doesn’t think his job is to take the pay cheque and be pleasant – he is there to bring about change. He hates bullshitters.”

He is also not an establishment type. The Spectator annual party is not his scene, even if by necessity he knows his way around the donor class.

He may be a disruptor, almost anarchist, but he also takes public service seriously. “When working in government for the first time he he taught himself maths to A-level and then degree level in order to properly process info and understand concepts of uncertainty in modelling. He’s also behind the push for maths in specialist schools,” says another friend.

“He brought in physicists to help Vote Leave properly crunch data. Such people will be involved in his plan for any general election. He will via Vote Leave have a huge bank of very helpful data of how people said they would vote in the referendum and how they did vote and therefore how they could behave in the next general election.”

It is too early to say whether Cummings is a short-order chef – contracted to get the UK out of the EU and then provide Johnson with a parliamentary majority – or is inside No 10 for a long haul. But even if it is only the former, he will have changed the UK forever.