The UK’s intelligence agencies were aware that Donald Trump may have been compromised by the Kremlin but Theresa May’s government – including then-foreign secretary Boris Johnson – chose not to pursue the matter for fear of offending the US president, a new book claims.

According to Crime in Progress, to be published next week in the US and UK, MI6 was “already aware” in 2016 of “suspicious links” between Russia and the Trump campaign. The book is written by Glenn Simpson and Peter Fritsch, who co-founded the Washington research firm Fusion GPS.

The firm has been at the centre of furious controversy in the US since the publication of Christopher Steele’s dossier of allegations about Trump and Russia.

The report was commissioned by the pair and Steele is quoted extensively in their new book. He describes Russians as “formidable adversaries” with a gift for deception. “It is perilous to underestimate them,” he says.

Citing his source network, Steele alleged in his dossier that Vladimir Putin’s regime had been cultivating Trump for at least five years and had collected enough compromising material to blackmail him – including from a Moscow hotel room. Trump has always denied these claims.

Steele’s conclusion was so alarming that he alerted the FBI and the British government. He briefed Sir Charles Farr, chairman of the joint intelligence committee, in the days after Trump’s unexpected victory, according to the new book. Steele is quoted as saying Farr took the dossier “extremely seriously”. Farr briefed Andrew Parker, then the head of MI5, and the “wider cabinet”, the book claims.

In December 2016 Steele also met Sir Richard Dearlove, the former head of MI6 and Steele’s old boss, at the Garrick Club, Simpson and Fritsch write. Steele “walked” Dearlove through the dossier’s explosive contents. Dearlove reportedly found the reporting “credible”.

Christopher Steele’s dossier on Trump included claims that the Kremlin had obtained potentially blackmailable material from a Moscow hotel room. Photograph: Victoria Jones/PA

The book claims: “[Dearlove] then surprised Steele by indicating that he was already aware that the British government had suspicions about links between Russia and members of the Trump campaign. It seemed that the British government had made a political decision not to push the matter further”.

It adds: “This news irritated Steele while at the same time reinforcing his view that the reporting was strong. If MI6 had reason to believe that the incoming head of state of Britain’s top ally could, in fact, be compromised by Moscow, he asked himself, why would the government be willing to ‘kick it into the long grass’.”

At this point Boris Johnson was foreign secretary. Johnson has robustly defended President Trump – including when the British ambassador in Washington, Sir Kim Darroch, was forced to resign following the leak of embarrassing diplomatic cables calling the White House “dysfunctional”.

Since becoming prime minister, Johnson has claimed there is no evidence Russia has ever meddled in British affairs.

On the election trail this week, Johnson again insisted there was nothing unusual about his refusal to publish a report by parliament’s intelligence and security committee on Russian interference in UK democracy. The report considers the flow of Moscow-linked money into the Conservative party, it is understood. Steele was one of several expert witnesses who gave evidence to the committee.

Contrary to Johnson’s view, Crime in Progress claims Russia is seeking to destabilise European politics in the same way as it did in the US. In April, the US special counsel Robert Mueller confirmed “sweeping and systematic” interference by Putin in the 2016 American presidential election. Mueller’s report did not find sufficient evidence that the Trump campaign “coordinated or conspired with the Russian government in its election-interference activities”.

Moscow may have boosted the Brexit cause in a “comparable way” during the EU referendum, the book suggests.

The book details how, before writing his dossier, Steele carried out a separate investigation into Russian intelligence activities in western Europe, codenamed Project Charlemagne. It focused on France, Italy, the UK, Germany and Turkey. He found that the Kremlin had “a secret black budget of tens of millions of dollars”, earmarked for populist nationalist politicians who were opposed to the EU, the new book claims.

Speaking to the Guardian last week, Simpson said Russian influence had wholly “compromised” the British political establishment. “There is a wilful blindness and determination to look the other way. There is a general awareness that Russian money has been pouring into the British political system. No-one in power wants to have that conversation,” he warned.

Simpson, a former Wall Street Journal reporter who specialised in covering Russian corruption, said that during the cold war Soviet spies had a different focus, trying to steal well-protected intelligence and defence secrets. Today’s Kremlin, by contrast, has found that the political system in western democracies “really is the soft underbelly”, he said.

The Kremlin’s UK playbook follows a similar pattern seen elsewhere in Europe, he added. It features Russian émigrés, or the children of émigrés, who make large donations to domestic political parties.

There was an “overlap” between Trump and Brexit involving many of the same characters and players such as former Trump adviser Steve Bannon, disgraced political consultants Cambridge Analytica and Brexit party leader Nigel Farage, Simpson said. Citing’s Fusion’s recent research, Simpson said Russian trolls that aggressively backed Brexit were also involved in supporting separatism in Catalonia and the French far-right.

Crime in Progress gives a detailed account of one of the enduring mysteries of the Trump-Russia story: who leaked Steele’s dossier to BuzzFeed News. It was David Kramer, a US former assistant secretary of state and longtime aide to the late Republican senator John McCain, as he has testified. Kramer saw Steele in London and got a copy of the dossier from Simpson.

Kramer wasn’t supposed to leak it, the book states. He was so appalled at Trump’s presidency, however, he took matters into his own hands. In January 2017 he met the BuzzFeed reporter Ken Bensinger in a deserted office. Kramer laid Steele’s memos out on the table and then excused himself, saying he had to make a phone call and go to the bathroom. Bensinger photographed the documents with his iPhone.

According to the book, Kramer also gave the memos to Carl Bernstein, the legendary Watergate reporter who was working for CNN. The dossier infuriated Trump, who denounces it still. “In journalism and politics, the old ‘I left the room for 20 minutes’ is a familiar ruse by sources who want plausible deniability for sharing confidential information,” the book points out.

Contacted by the Guardian, Dearlove rubbished the account of his meeting. He said: “If I had retired in 2004 how would I be likely to know what the inner thinking of the British government was on a very sensitive issue in 2016?”

A government spokesperson said: “Investigations in the US are a matter for the US authorities. We won’t comment on the ongoing investigations.”

• Crime in Progress: The Secret History of the Trump-Russia Investigation is published on 26 November by Allen Lane