Moscow, summer 1991. Mikhail Gorbachev is in power. Official relations with the west have softened, but the KGB still assumes all western embassy workers are spooks. The KGB agents assigned to them are easy to spot. They have a method. Sometimes they pursue targets on foot, sometimes in cars. The officers charged with keeping tabs on western diplomats are never subtle.

One of their specialities is breaking into Moscow apartments. The owners are always away, of course. The KGB leave a series of clues – stolen shoes, women’s tights knotted together, cigarette butts stomped out and left demonstratively on the floor. Or a surprise turd in the toilet, waiting in grim ambush. The message, crudely put, is this: we are the masters here! We can do what the fuck we please!

Back then, the KGB kept watch on all foreigners, especially American and British ones. The UK mission in Moscow was under close observation. The British embassy was a magnificent mansion built in the 1890s by a rich sugar merchant, on the south bank of the Moskva river. It looked directly across to the Kremlin. The view was dreamy: a grand palace, golden church domes and medieval spires topped with revolutionary red stars.

One of those the KGB routinely surveilled was a 27-year-old diplomat, newly married to his wife, Laura, on his first foreign posting, and working as a second secretary in the chancery division. In this case, their suspicions were right.

The “diplomat” was a British intelligence officer. His workplace was a grand affair: chandeliers, mahogany-panelled reception rooms, gilt-framed portraits of the Queen and other royals hanging from the walls. His desk was in the embassy library, surrounded by ancient books. The young officer’s true employer was an invisible entity back in London – SIS, the Secret Intelligence Service, commonly known as MI6.

His name was Christopher Steele. Years later, he would be commissioned to undertake an astonishing secret investigation. It was an explosive assignment: to uncover the Kremlin’s innermost secrets with relation to Donald Trump. Steele’s findings, and the resulting dossier, would shake the American intelligence community and cause a political earthquake not seen since the dark days of Richard Nixon and Watergate.

Steele had arrived in Moscow via the usual establishment route for upwardly mobile British spies: the University of Cambridge. Cambridge had produced some of MI6’s most talented cold war officials. A few of them, it turned out – to great embarrassment – had secret second jobs with the KGB. The joke inside MI6 was that only those who had never visited the Soviet Union would wish to defect.

Steele had studied social and political sciences at Girton College. His views were centre-left; he and his elder sister were the first members of his family to go to university. (Steele’s paternal grandfather was a coal miner from Pontypridd in south Wales; his great-uncle died in a pit accident.) Steele wrote for the student newspaper, Varsity. He became president of the Cambridge Union, a debating society dominated by well-heeled and well-connected young men and women.

It’s unclear who recruited Steele. Traditionally, certain Cambridge tutors were rumoured to identify promising MI6 candidates. Whatever the route, Steele’s timing was good. After three years at MI6, he was sent to the Soviet Union in April 1990, soon after the fall of the Berlin Wall and the collapse of the communist bloc across eastern Europe.

It was a tumultuous time. Seventy years after the Bolshevik revolution, the red empire was crumbling. The Baltic states had revolted against Soviet power; their own national authorities were governing in parallel with Moscow. In June 1991, the Russian Soviet Federative Socialist Republic elected a democratic president, Boris Yeltsin. Food shortages were not uncommon.

There was still much to enjoy. Like other expatriates, the Steeles visited the Izmailovsky craft market, next to an imperial park where Peter the Great’s father, Tsar Alexei, had established a model farm. Here you could buy lacquered boxes, patchwork quilts, furry hats and Soviet kitsch. Steele acquired samovars, carpets from central Asia, a papier-mache Stalin mask and a hand-painted Tolstoy doll set.

Much of the Soviet Union was off-limits to diplomats. Steele was the embassy’s “internal traveller”. He visited newly accessible cities. One of them was Samara, a wartime Soviet capital. There, he became the first foreigner to see Stalin’s underground bunker. Instead of Lenin, he found dusty portraits of Peter the Great and the imperial commander Mikhail Kutuzov – proof, seemingly, that Stalin was more nationalist than Marxist. Another city was Kazan, in Tatarstan. There a local correspondent, Anatoly Andronov, took a black-and-white photo of Steele chatting with newspaper editors. At weekends, Steele took part in soccer matches with a group of expats in a Russian league. In one game, he played against the legendary Soviet Union striker Oleh Blokhin, who scored from the halfway line.

Christopher Steele in early 1991, with newspaper editors in the Tatar city of Kazan. Photograph: Anatoly Andronov

The atmosphere was optimistic. It seemed to Steele that the country was shifting markedly in the right direction. Citizens once terrified of interacting with outsiders were ready to talk. The KGB, however, found nothing to celebrate in the USSR’s tilt towards freedom and reform. In August 1991, seven apparatchiks staged a coup while Gorbachev was vacationing in Crimea. Most of the British embassy was away. Steele was home at his second-floor apartment in Gruzinsky Pereulok. He left the apartment block and walked for 10 minutes into town. Crowds had gathered outside the White House, the seat of government; thus far the army hadn’t moved against them.

From 50 yards away, Steele watched as a snowy-haired man in a suit climbed on a tank and – reading from notes brushed by the wind – denounced the coup as cynical and illegal. This was a defiant Yeltsin. Steele listened as Yeltsin urged a general strike and, fist clenched, told his supporters to remain strong.

The coup failed, and a weakened Gorbachev survived. The putschists – the leading group in all the main Soviet state and party institutions – were arrested. In the west, and in the US in particular, many concluded that Washington had won the cold war, and that, after decades of ideological struggle, liberal democracy had triumphed.

Steele knew better. Three days after the coup, surveillance on him resumed. His MI6 colleagues in Hungary and Czechoslovakia reported that after revolutions there the secret police vanished, never to come back. But here were the same KGB guys, with the same familiar faces. They went back to their old routines of bugging, break-ins and harassment.

The regime changed. The system didn’t.

By the time Steele left Moscow in April 1993, the Soviet Union had gone. A new country, led by Yeltsin, had replaced it: the Russian Federation. The KGB had been dissolved, but its officers hadn’t exactly disappeared. They still loathed the US and were merely biding their time.

One mid-ranking former KGB spy who was unhappy about this state of affairs was Vladimir Putin. Putin had been posted to Dresden in provincial East Germany in the mid-80s, and had missed perestroika and glasnost, Gorbachev’s reformist ideas. He had now returned to the newly renamed St Petersburg and was carving out a political career. He mourned the end of the USSR, and once called its disappearance “the greatest geopolitical catastrophe of the 20th century”.

A post-communist spy agency, the Federal Security Service, or FSB, had taken over the KGB’s main functions. Back in the UK, Steele would soon move into MI6’s purpose-built new office – a large, striking, postmodern pile of a building overlooking the River Thames in London. Staff called it Vauxhall Cross. This gaudy Babylonian temple was hard to miss; in 1994, the government officially acknowledged the existence of MI6 for the first time. The FSB would become its bitter adversary.

From London, Steele continued to work on the new Russia. He was ambitious, keen to succeed, and keen to be seen to succeed. He was also, perhaps, less posh than some of his upper-class peers. Steele’s father, Perris, and his mother, Janet, met when they worked together at the UK Met Office. Perris was a forecaster for the armed services. The family had lived on army bases in Aden, where Steele was born, on the Shetland Islands (where he found an interest in bird-watching) and – twice – in Cyprus.

Steele’s education had been varied. He went to a British forces school in Cyprus. He did sixth form at a college in Berkshire. He then spent a “seventh” or additional term at Wellington College, an elite private boarding school. There he sat the entrance exam for Cambridge.

At MI6, Steele moved in a small world of Kremlin specialists. There were conferences and seminars in university towns like Oxford; contacts to be made; émigrés to be met, lunched and charmed. In 1998 he got another posting, to the British embassy in Paris. He had a family: two sons and a daughter, born in France, where Steele was officially First Secretary Financial.

At this point, his career hit a bump. In 1999, a list of MI6 officers was leaked online. Steele was one of them. He appeared as “Christopher David Steele, 90 Moscow; dob 1964”.

The breach wasn’t Steele’s fault, but it had unfortunate consequences. As an exposed British officer, he couldn’t go back to Russia.

In Moscow, the spies were staging a comeback. In 1998 Putin became FSB chief, then prime minister, and in 2000, president. By 2002, when Steele left Paris, Putin had consolidated his grip. Most of Russia’s genuine political opposition had been wiped out, from parliament as well as from public life and the evening news. The idea that Russia might slowly turn into a democracy had proved a late-century fantasy. Rather, the US’s traditional nuclear-armed adversary was moving in an authoritarian direction.

At first, George W Bush and Tony Blair viewed Putin as a respectable ally in the war against terror. But he remained an enigma. As Steele knew better than most, obtaining information from inside the presidential administration in Moscow was tough. One former member of the US National Security Council described Putin as a “black box”. “The Brits had slightly better assets than us. We had nothing. No human intelligence,” the source said. And, with the focus on fighting Islamists, Russia was downgraded on the list of US-UK intelligence priorities.

By 2006, Steele held a senior post at MI6’s Russia desk in London. There were ominous signs that Putin was taking Russia in an aggressive direction. The number of hostile Russian agents in the UK grew, surpassing cold war levels. Steele tracked a new campaign of subversion and covert influence.

And then two FSB assassins put a radioactive poison into the tea of Alexander Litvinenko, a former FSB officer turned London-based dissident. It was an audacious operation, and a sign of things to come. MI6 picked Steele to investigate. One reason for this was that he wasn’t emotionally involved with the case, unlike some of his colleagues who had known the victim. He quickly concluded the Russian state had staged the execution.

Alexander Litvinenko, whose poisoning in London in 2006 Christopher Steele was chosen to investigate. Photograph: Natasja Weitsz

Steele’s gloomy view of Russia – that under Putin it was not only domestically repressive but also internationally reckless and revisionist – looked about right. Steele briefed government ministers. Some got it. Others could scarcely believe Russian spies would carry out murder and mayhem on the streets of London.

All told, Steele spent 22 years as a British intelligence officer. There were some high points – he saw his years in Moscow as formative – and some low ones. Two of the diplomats with whom he shared an office in the embassy library, Tim Barrow and David Manning, went on to become UK ambassadors to the EU and the US respectively.

Steele didn’t quite rise to the top, in what was a highly competitive service. Espionage might sound exciting, but the salary of a civil servant was ordinary. And in 2009 he had faced a personal tragedy, when his wife died at the age of 43 after a period of illness.

That same year, Steele left MI6 and set up his own business intelligence firm, Orbis, in partnership with another former British spy, Christopher Burrows. The transition from government to the private sector wasn’t easy. Steele and Burrows were pursuing the same intelligence matters as before, but without the support and peer review they had in their previous jobs. MI6’s security branch would often ask an officer to go back to a source, or redraft a report, or remark: “We think it’s interesting. We’d like to have more on this.” This kept up quality and objectivity.

Steele and Burrows, by contrast, were out on their own, where success depended more on one’s own wits. There was no more internal challenge. The people they had to please were corporate clients. The pay was considerably better.

The shabby environs of Orbis’s office in London’s Victoria, where I first met Christopher Steele, were a long way away from Washington DC and the bitterly contested 2016 US presidential election. So how did Steele come to be commissioned to research Donald J Trump and produce his devastating dossier?

At the same moment Steele said goodbye to official spying, another figure was embarking on a new career in the crowded field of private business intelligence. His name was Glenn Simpson. He was a former journalist. Simpson was an alluring figure: a large, tall, angular, bear-like man who slotted himself easily on to a bar stool and enjoyed a beer or two. He was a good-humoured social companion who spoke in a nasal drawl. Behind small, oval glasses was a twinkling intelligence. He excelled at what he did.

Simpson had been an illustrious Wall Street Journal correspondent. Based in Washington and Brussels, he had specialised in post-Soviet murk. He didn’t speak Russian or visit the Russian Federation. This was deemed too dangerous. Instead, from outside the country, he examined the dark intersection between organised crime and the Russian state.

By 2009, Simpson decided to quit journalism, at a time when the media industry was in all sorts of financial trouble. He co-founded his own commercial research and political intelligence firm, based in Washington DC. Its name was Fusion GPS. Its website gave little away. It didn’t even list an address.

Simpson then met Steele. They knew some of the same FBI people and shared expertise on Russia. Fusion and Orbis began a professional partnership. The Washington- and London-based firms worked for oligarchs litigating against other oligarchs. This might involve asset tracing – identifying large sums concealed behind layers of offshore companies.

Later that year, Steele embarked on a separate, sensitive new assignment that drew on his knowledge of covert Russian techniques – and of football. (In Moscow he had played at full-back.) The client was the English Football Association, the FA. England was bidding to host the 2018 soccer World Cup. Its main rival was Russia. There were joint bids, too, from Spain and Portugal, and the Netherlands and Belgium. His brief was to investigate the eight other bidding nations, with a particular focus on Russia. It was rumoured that the FSB had carried out a major influence operation, ahead of a vote in Zurich by the executive committee of Fifa, soccer’s international governing body.

Steele discovered that Fifa corruption was global. It was a stunning conspiracy. He took the unusual step of briefing an American contact in Rome, the head of the FBI’s Eurasian serious crime division. This “lit the fuse”, as one friend put it, and led to a probe by US federal prosecutors. And to the arrest in 2015 of seven Fifa officials, allegedly connected to $150m (£114m) in kickbacks, paid on TV deals stretching from Latin America to the Caribbean. The US indicted 14 individuals.

The episode burnished Steele’s reputation inside the US intelligence community and the FBI. Here was a pro, a well-connected Brit, who understood Russian espionage and its subterranean tricks. Steele was regarded as credible. Between 2014 and 2016, Steele authored more than 100 reports on Russia and Ukraine. These were written for a private client but shared widely within the US state department, and sent up to secretary of state John Kerry and assistant secretary of state Victoria Nuland, who was in charge of the US response to Putin’s annexation of Crimea and covert invasion of eastern Ukraine. Many of Steele’s secret sources were the same people who would later supply information on Trump.

One former state department envoy during the Obama administration said he read dozens of Steele’s reports. On Russia, the envoy said, Steele was “as good as the CIA or anyone”.

Steele’s professional reputation inside US agencies would prove important the next time he discovered alarming material.

Trump’s political rise in the autumn of 2015 and the early months of 2016 was swift and irresistible. The candidate was a human wrecking ball who flattened everything in his path, including the Republican party’s aghast, frozen-to-the-spot establishment. Marco Rubio, Jeb Bush, Ted Cruz – all were batted aside, taunted, crushed. Scandals that would have killed off a normal presidential candidate made Trump stronger. The media loved it. Increasingly, so did the voters. Might anything stop him?

In mid-2015, the Republican front-runner had been Jeb Bush, son of one US president and brother of another. But as the campaign got under way, Bush struggled. Trump dubbed the former Florida governor “low-energy”. During the primaries, a website funded by one of Trump’s wealthy Republican critics, Paul Singer, commissioned Fusion to investigate Trump.

After Trump became the presumptive nominee in May 2016, Singer’s involvement ended and senior Democrats seeking to elect Hillary Clinton took over the Trump contract. The new client was the Democratic National Committee. A lawyer working for Clinton’s campaign, Marc E Elias, retained Fusion and received its reports. The world of private investigation was a morally ambiguous one – a sort of open market in dirt. Information on Trump was of no further use to Republicans, but it could be of value to Democrats, Trump’s next set of opponents.

Before this, in early spring 2016, Simpson approached Steele, his friend and colleague. Steele began to scrutinise Paul Manafort, who would soon become Trump’s new campaign manager. From April, Steele investigated Trump on behalf of the DNC, Fusion’s anonymous client. All Steele knew at first was that the client was a law firm. He had no idea what he would find. He later told David Corn, Washington editor of the magazine Mother Jones: “It started off as a fairly general inquiry.” Trump’s organisation owned luxury hotels around the world. Trump had, as far back as 1987, sought to do real estate deals in Moscow. One obvious question for him, Steele said, was: “Are there business ties to Russia?”

Paul Manafort, who Steele started investigating in spring 2016. Last month Manafort was indicted on 12 charges including conspiracy against the United States. Photograph: Matt Rourke/AP

Over time, Steele had built up a network of sources. He was protective of them: who they were he would never say. It could be someone well-known – a foreign government official or diplomat with access to secret material. Or it could be someone obscure – a lowly chambermaid cleaning the penthouse suite and emptying the bins in a five-star hotel.

Normally an intelligence officer would debrief sources directly, but since Steele could no longer visit Russia, this had to be done by others, or in third countries. There were intermediaries, subsources, operators – a sensitive chain. Only one of Steele’s sources on Trump knew of Steele. Steele put out his Trump-Russia query and waited for answers. His sources started reporting back. The information was astonishing; “hair-raising”. As he told friends: “For anyone who reads it, this is a life-changing experience.”

Steele had stumbled upon a well-advanced conspiracy that went beyond anything he had discovered with Litvinenko or Fifa. It was the boldest plot yet. It involved the Kremlin and Trump. Their relationship, Steele’s sources claimed, went back a long way. For at least the past five years, Russian intelligence had been secretly cultivating Trump. This operation had succeeded beyond Moscow’s wildest expectations. Not only had Trump upended political debate in the US – raining chaos wherever he went and winning the nomination – but it was just possible that he might become the next president. This opened all sorts of intriguing options for Putin.

In June 2016, Steele typed up his first memo. He sent it to Fusion. It arrived via enciphered mail. The headline read: US Presidential Election: Republican Candidate Donald Trump’s Activities in Russia and Compromising Relationship with the Kremlin. Its text began: “Russian regime has been cultivating, supporting and assisting TRUMP for at least 5 years. Aim, endorsed by PUTIN, has been to encourage splits and divisions in the western alliance.”

“So far TRUMP has declined various sweetener real estate business deals, offered him in Russia to further the Kremlin’s cultivation of him. However he and his inner circle have accepted a regular flow of intelligence from the Kremlin, including on his Democratic and other political rivals.

“Former top Russian intelligence officer claims FSB has compromised TRUMP through his activities in Moscow sufficiently to be able to blackmail him. According to several knowledgeable sources, his conduct in Moscow has included perverted sexual acts which have been arranged/monitored by the FSB.

“A dossier of compromising material on Hillary CLINTON has been collated by the Russian Intelligence Services over many years and mainly comprises bugged conversations she had on various visits to Russia and intercepted phone calls rather than any embarrassing conduct. The dossier is controlled by Kremlin spokesman, PESKOV, directly on Putin’s orders. However, it has not yet been distributed abroad, including to TRUMP. Russian intentions for its deployment still unclear.”

The memo was sensational. There would be others, 16 in all, sent to Fusion between June and early November 2016. At first, obtaining intelligence from Moscow went well. For around six months – during the first half of the year – Steele was able to make inquiries in Russia with relative ease. It got harder from late July, as Trump’s ties to Russia came under scrutiny. Finally, the lights went out. Amid a Kremlin cover-up, the sources went silent and information channels shut down.

If Steele’s reporting was to be believed, Trump had been colluding with Russia. This arrangement was transactional, with both sides trading favours. The report said Trump had turned down “various lucrative real estate development business deals in Russia”, especially in connection with the 2018 World Cup, hosted by Moscow. But he had been happy to accept a flow of Kremlin-sourced intelligence material, apparently delivered to him by his inner circle. That didn’t necessarily mean the candidate was a Russian agent. But it did signify that Russia’s leading spy agency had expended considerable effort in getting close to Trump – and, by extension, to his family, friends, close associates and business partners, not to mention his campaign manager and personal lawyer.

On the eve of the most consequential US election for generations, one of the two candidates was compromised, Steele’s sources claimed. The memo alleged that Trump had unusual sexual proclivities, and that the FSB had a tape. If true, this meant he could indeed be blackmailed.

When I met Steele in December 2016, he gave no hint he had been involved in what was the single most important investigation in decades.

Steele’s collaborators offered salacious details. The memo said that Russian intelligence had sought to exploit “TRUMP’s personal obsessions and sexual perversion” during his 2013 stay at Moscow’s Ritz-Carlton hotel for the Miss Universe beauty pageant. The operation had allegedly worked. The tycoon had booked the presidential suite of the Ritz-Carlton hotel “where he knew President and Mrs OBAMA (whom he hated) had stayed on one of their official trips to Russia”.

There, the memo said, Trump had deliberately “defiled” the Obamas’ bed. A number of prostitutes “had performed a ‘golden showers’ (urination) show in front of him”. The memo also alleged: “The hotel was known to be under FSB control with microphones and concealed cameras in all the main rooms to record anything they wanted to.”

As well as sex, there was another fascinating dimension to this alleged plot, categorically denied by Trump. According to Steele’s sources, associates of Trump had held a series of clandestine meetings in central Europe, Moscow and elsewhere with Russian spies. The Russians were very good at tradecraft. Nonetheless, could this be a trail that others might later detect?

Steele’s sources offered one final devastating piece of information. They alleged that Trump’s team had co-ordinated with Russia on the hacking operation against Clinton. And that the Americans had secretly co-paid for it.

Donald Trump and Gabriela Isler, winner of Miss Universe 2013, in Moscow. Photograph: Kommersant/Getty

Steele wrote up his findings in MI6 house style. The memos read like CX reports – classified MI6 intelligence documents. They were marked “confidential/sensitive source”. The names of prominent individuals were in caps – TRUMP, PUTIN, CLINTON. The reports began with a summary. They offered supporting detail. Sources were anonymous. They were introduced in generic terms: “a senior Russian foreign ministry figure” or “a former top level Russian intelligence officer still active inside the Kremlin”. They were given letters, starting with A and proceeding down the alphabet.

How certain was Steele that his sources had got it right and that he wasn’t being fed disinformation? The matter was so serious, so important, so explosive, so far-reaching, that this was an essential question.

As spies and former spies knew, the world of intelligence was non-binary. There were degrees of veracity. A typical CX report would include phrases such as “to a high degree of probability”. Intelligence could be flawed, because humans were inherently unreliable. They forgot things. They got things wrong.

One of Steele’s former Vauxhall Cross colleagues likened intelligence work to delicate shading. This twilight world wasn’t black and white; it was a muted palette of greys, off-whites and sepia tones, he told me. He said you could shade in one direction – more optimistically – or in another direction – less optimistically. Steele was generally in the first category.

Steele was adamant that his reporting was credible. One associate described him as sober, cautious, highly regarded, professional and conservative. “He’s not the sort of person who will pass on gossip. If he puts something in a report, he believes there is sufficient credibility in it,” the associate said. The idea that Steele’s work was fake or a cowboy operation or born of political malice was completely wrong, he added.

The dossier, Steele told friends, was a thoroughly professional job, based on sources who had proven themselves in other areas. Evaluating sources depended on a critical box of tools: what was a source’s reporting record, was he or she credible, what was the motivation?

Steele recognised that no piece of intelligence was 100% right. According to friends, he assessed that his work on the Trump dossier was 70-90% accurate. Over eight years, Orbis had produced scores of reports on Russia for private clients. A lot of this content was verified or “proven up”. As Steele told friends: “I’ve been dealing with this country for 30 years. Why would I invent this stuff?”

In late 2015 the British eavesdropping agency, GCHQ, was carrying out standard “collection” against Moscow targets. These were known Kremlin operatives already on the grid. Nothing unusual here – except that the Russians were talking to people associated with Trump. The precise nature of these exchanges has not been made public, but according to sources in the US and the UK, they formed a suspicious pattern. They continued through the first half of 2016. The intelligence was handed to the US as part of a routine sharing of information.

The FBI and the CIA were slow to appreciate the extensive nature of these contacts between Trump’s team and Moscow. This was in part due to institutional squeamishness – the law prohibits US agencies from examining the private communications of US citizens without a warrant.

But the electronic intelligence suggested Steele was right. According to one account, the US agencies looked as if they were asleep. “‘Wake up! There’s something not right here!’ – the BND [German intelligence], the Dutch, the French and SIS were all saying this,” one Washington-based source told me.

That summer, GCHQ’s then head, Robert Hannigan, flew to the US to personally brief CIA chief John Brennan. The matter was deemed so important that it was handled at “director level”, face-to-face between the two agency chiefs. James Clapper, director of national intelligence, later confirmed the “sensitive” stream of intelligence from Europe. After a slow start, Brennan used the GCHQ information and other tip-offs to launch a major inter-agency investigation. Meanwhile, the FBI was receiving disturbing warnings from Steele.

At this point, Steele’s Fusion material was unpublished. Whatever the outcome of the election, it raised grave questions about Russian interference and the US democratic process. There was, Steele felt, overwhelming public interest in passing his findings to US investigators. The US’s multiple intelligence agencies had the resources to prove or disprove his discoveries. He realised that these allegations were, as he put it to a friend, a “radioactive hot potato”. He anticipated a hesitant response, at least at first.

In June, Steele flew to Rome to brief the FBI contact with whom he had co-operated over Fifa. His information started to reach the bureau in Washington. It had certainly arrived by the time of the Democratic National Convention in late July, when WikiLeaks first began releasing hacked Democratic emails. It was at this moment that FBI director James Comey opened a formal investigation into Trump-Russia.

Trump and Putin at the Apec summit in Vietnam this week. Photograph: Mikhail Klimentyev/TASS

In September, Steele went back to Rome. There he met with an FBI team. Their response was one of “shock and horror,” Steele said. The bureau asked him to explain how he had compiled his reports, and to give background on his sources. It asked him to send future copies.

Steele had hoped for a thorough and decisive FBI investigation. Instead, it moved cautiously. The agency told him that it couldn’t intervene or go public with material involving a presidential candidate. Then it went silent. Steele’s frustrations grew.

Later that month, Steele had a series of off-the-record meetings with a small number of US journalists. They included the New York Times, the Washington Post, Yahoo! News, the New Yorker and CNN. In mid-October he visited New York and met with reporters again.

Comey then announced he was reopening an investigation into Clinton’s use of a private email server. At this point, Steele’s relationship with the FBI broke down. The excuse given by the bureau for saying nothing about Trump looked bogus. In late October, Steele spoke to the Mother Jones editor David Corn via Skype.

The story was of “huge significance, way above party politics”, Steele said. He believed Trump’s Republican colleagues “should be aware of this stuff as well”. Of his own reputation, Steele said: “My track record as a professional is second to no one.” Steele acknowledged that his memos were works in progress, and was genuinely worried about the implications of the allegations. “The story has to come out,” he told Corn.

At this point Steele was still anonymous, a ghost. But the ghost’s message was rapidly circulating on Capitol Hill and inside Washington’s spy agencies, as well as among certain journalists and thinktanks. Democratic senators now apprised of Steele’s work were growing exasperated. The FBI seemed unduly keen to trash Clinton’s reputation while sitting on explosive material concerning Trump.

One of those who was aware of the dossier’s broad allegations was the Senate minority leader, Harry Reid, a Democrat. In August Reid, had written to Comey and asked for an inquiry into the “connections between the Russian government and Donald Trump’s presidential campaign”. In October, Reid wrote to Comey again. This time he framed his inquiry in scathing terms. In a clear reference to Steele, Reid wrote: “In my communications with you and other top officials in the national security community, it has become clear that you possess explosive information about close ties and coordination between Donald Trump, his top advisors and the Russian government … The public has a right to know this information.”

But all this frantic activity came to nought. Just as Nixon was re-elected during the early stages of Watergate, Trump won the presidential election, to general dismay, at a time when the Russia scandal was small but growing. Steele had found prima facie evidence of a conspiracy, but by and large the US public knew nothing about it. In November, his dossier began circulating in the top national security echelons of the Obama administration. But it was too late.

The same month a group of international experts gathered in Halifax on Canada’s eastern seaboard. Their task: to make sense of the world in the aftermath of Trump’s stunning victory. One of the delegates attending the Halifax International Security Forum was Senator John McCain. Another was Sir Andrew Wood, the UK’s former ambassador to Russia. Wood was a friend of Steele’s and an Orbis associate. Before the election, Steele had gone to Wood and shown him the dossier. He wanted the ambassador’s advice. What should he do, or not do, with it? Of the dossier, Wood told me: “I took it seriously.”

On the margins of the Halifax conference, Wood briefed McCain about Steele’s dossier – its contents, if true, had profound and obvious implications for the incoming Trump administration, for the Republican party, and for US democracy. The implications were alarming enough to lead McCain to dispatch a former senior US official to meet Steele and find out more.

Christopher Steele in London in March. Photograph: Victoria Jones/PA

The emissary was David Kramer, a former assistant secretary of state in the Bush administration. He was sufficiently troubled to get on a flight to London. Steele agreed to meet him at Heathrow airport. The rendezvous involved some old-fashioned spycraft. Kramer didn’t know what Steele looked like. He was told to look for a man with a copy of the Financial Times. After meeting Kramer, Steele drove him to his home in Surrey. They talked through the dossier: how Steele compiled it, what it said. Less than 24 hours later, Kramer returned to Washington. Glenn Simpson then shared a copy of the dossier confidentially with McCain, along with a final Steele memo on the Russian hacking operation, written in December.

McCain believed it was impossible to verify Steele’s claims without a proper investigation. He made a call and arranged a meeting with Comey. Their encounter on 8 December 2016 lasted five minutes. Not much was said. McCain gave Comey the dossier.

McCain’s intervention now made some kind of bureaucratic response inevitable. This was no longer just an FBI affair; it required co-ordination across the top levels of US intelligence. A highly classified two-page summary of Steele’s dossier was compiled. It was attached to a longer, restricted briefing note on Russian cyber interference in the 2016 election. The US’s most senior intelligence chiefs mulled what to do.

Their next task was an unenviable one. As former CIA director Michael Hayden put it to me, the situation was “off the map in terms of what intelligence is asked to do. I didn’t envy them”. Of the dossier, Hayden said: “My gestalt idea, when I saw it, was that this looks like our stuff.”

The dossier was on its way to the desk of the man who was still, for now, the world’s most powerful person: President Barack Obama.

It was also going to his successor, the next guy in the Oval Office. He wasn’t going to like it much.

Main illustration by Nate Kitch

This is an edited extract from Collusion by Luke Harding, which is published by Guardian Faber at £14.99. To order a copy for £9.99, go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders minimum p&p of £1.99. The book is available to purchase in Australia for $29.99.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.