In 2003, Rudolph Giuliani made his first trip to Ukraine to lay a symbolic capsule at the site of a new memorial to the victims of 9/11, the tragedy from which he had emerged as “America’s mayor.”

The ceremony, organised by a local businessman and politician named Vadim Rabinovich, was a modest affair attended by Kiev’s deputy mayor and diplomats from the US and Israel, but it marked the beginning of what would become a lucrative string of deals for Giuliani in the former Soviet state.

“I have met many young and promising politicians here who are not indifferent to the fate of their government,” Giuliani told local reporters. “I have understood how hospitable and friendly the people of Ukraine are, how they honestly strive to reach out to Americans, how much they want to oppose violence and terror.”

Sixteen years later, Rudolph Giuliani’s labyrinthine ties to Ukraine have emerged as a central element in the impeachment inquiry against his client, Donald Trump, as details emerge of what one senior diplomat called a “shadow foreign policy” to trigger an investigation into Joe Biden.

The following account is based on interviews with associates of Giuliani and his business partners in Ukraine, prosecutors familiar with his attempts to gather information about the Bidens, and officials and politicians concerned about the flamboyant former mayor.

The Guardian’s reporting shows that Giuliani boasted of his ties to Trump to gain access to high-ranking prosecutors and officials while his associates used the Giuliani name to bolster their reputations and legitimise dubious business ventures.

“He announced that he was a lawyer for the president of the United States, that he was working for Donald Trump, there was nothing to hide,” said Andrii Telizhenko, a former Ukrainian embassy official who met with Giuliani in the US over burgers and cigars.

Giuliani’s ties became a US political story when two Soviet-born American businessmen, Lev Parnas and Igor Fruman, helped connect Giuliani with powerful Ukrainian prosecutors who claimed to have information about a scandal tied to Biden’s son, Hunter, and in turn sought to trade in on Giuliani’s name to advance their own business interests.

The pair were arrested on charges of violating campaign finance laws while attempting to travel to Vienna, shortly after they met with Giuliani in Washington DC. Giuliani is also reportedly under investigation for whether his work in Ukraine broke laws on foreign lobbying.

A crucial contact for Giuliani was Yuriy Lutsenko, an embattled top prosecutor who faced being fired after the election of the Ukrainian president, Volodymyr Zelenskiy.

Lutsenko, who has himself been accused of corruption by civil society activists, said he met Giuliani because he was ready to discuss a joint official investigation into Burisma – the gas company where Hunter Biden worked – and also to pursue a separate investigation into money stolen by the previous government.

Lutsenko was one of five prosecutors contacted by Giuliani, including the former general prosecutor Viktor Shokin, who resigned under pressure in 2016.

In Ukraine, Lutsenko was widely seen as an intriguer. Yegor Sobolev, a former politician and head of the Ukrainian parliament’s anti-corruption committee, said that he was “a new kind” of prosecutor, “not afraid to step across the line and get involved in politics” unlike his predecessors.

Lutsenko also clashed with the then US ambassador Marie Yovanovitch, who told congressional investigators earlier this month she had learned of a “concerted campaign” to have her fired. Yovanovitch was abruptly removed in May after Giuliani pressed for changes at the embassy.

According to the indictment against Fruman and Parnas, the campaign against her was led by a Ukrainian official, but Lutsenko denied he had asked Giuliani to lobby to have Yovanovitch fired.

“Giuliani and especially these two guys [Parnas and Fruman] told me that they are against her as an ambassador long before our meeting,” Lutsenko said in a telephone interview from London where he is attending English classes.

“Yes I had not a good relationship with her because she tried openly to control my activity and to interfere in our activity … but I never asked about any deal.”

Lutsenko said last year that Yovanovitch had given him a list of people he should not prosecute, but later retracted the claim. A law enforcement source has told the Guardian that Lutsenko exaggerated claims to Giuliani, including the “don’t prosecute” list, in order to curry political favour to help him keep his job.

Giuliani has since claimed, without evidence, that the “don’t prosecute” list was part of a liberal anti-Trump conspiracy bankrolled by the philanthropist George Soros

Also in Giuliani’s meetings with Lutsenko were Parnas and Fruman, two Florida real estate developers born in the former Soviet Union, who had become key operatives for Giuliani to dig up dirt in Ukraine. They had turned heads with a $325,000 donation to a Trump-aligned political action committee in May 2018, facilitating access to the White House and the US president’s Florida resort Mar-a-Lago.

They also paid Giuiliani $500,000, the former New York mayor has said, and accompanied him to high-profile events such as the state funeral of George Bush, the former US president, in December.

Fruman, 53, had the better connections in Ukraine. Since the 1990s, he had run a luxury goods business in Odessa, the Black Sea port city, that provided everything from Jaguar automobiles to exotic food products to Ukraine’s jet set, and he also owned stakes in hotels and Kyiv clubs including Buddha Bar.

But business could be unpredictable: SEC records show that Fruman’s company, Fd Import & Export Corp, sued in 2001 for nearly $1m in damages when a shipment of bananas from Ecuador to Ukraine was shown to be diseased and “fraudulently and intentionally” mislabeled as Bonita bananas.

By 2016 – even before a difficult divorce settlement with his ex-wife – Fruman was seen as desperate.

“He was just a bankrupt guy, a crook I would say, who was looking for some fortune and opportunities,” said David Sakvarelidze, a former Ukrainian regional prosecutor who met Fruman after he was assigned to work in Odessa in 2015. “He had big financial troubles and I was told he basically already was living in the United States at that time, in Miami. He was looking for some opportunities to settle his financial problems. Maybe he could use his connections to Giuliani’s team to settle his financial troubles, that’s my presumption.”

Fruman and Parnas appeared to do just that, attempting to persuade the Ukrainian oligarch Ihor Kolomoisky to set up a meeting with Zelenskiy, the new president, and also pitching a scheme to replace the head of Naftogaz, the Ukrainian state oil and gas company, with an executive who could direct business their way.

Rick Perry, the US energy secretary, later made similar efforts, lobbying the Ukrainian government to make changes to Naftogaz’s board of directors, the Associated Press reported.

Fruman also had become a major player in Kyiv’s Jewish community, employing Giuliani’s public profile to help fundraise for one of Ukraine’s top rabbis, Moshe Reuven Azman.

At a celebration for the 120th anniversary of the founding of Kyiv’s Brodsky synagogue, Fruman pledged to raise more than $1m for orphans at a village established by Azman for Jewish refugees from the war in east Ukraine. And in an interview with an Israeli journalist, Fruman claimed to have delivered a handwritten note from Azman to Trump. Azman declined repeated requests for interviews.

Also present at the synagogue was Rabinovich, the Ukrainian businessman, who donated the memorial in Kyiv to the victims of the September 11 terrorist attacks in New York bearing the words “thou shalt not kill”. Through his library foundation, Rabinovich claimed Giuliani had not been paid and said he had not seen Giuliani since the visit.

But the 2003 visit heralded a decade of consulting and publicity trips to Ukraine for Giuliani, who has traded on his US political connections and his reputation as the mayor who steered New York through the Trade Centre attack to collect a small fortune in lucrative speaking fees and security contracts in the ex-Soviet state.

In 2008, he consulted the former heavyweight boxer Vitali Klitschko during his campaign to become the mayor of Kyiv. In 2017, he spoke at a conference organised by Victor Pinchuk, the Ukrainian metals magnate who had previously donated $150,000 to Trump’s charitable foundation, attracting scrutiny from the special prosecutor, Robert Mueller.

His security firm was also hired by Pavel Fuks, a wealthy businessman from Kharkiv, in 2017 to consult the city on its emergency services and foreign investment. While the value of the contracts have not been made public, a person with knowledge of Giuliani’s business dealings said their total value was well above $1m.

For many Ukrainians, Giuliani was seen as a bridge into exclusive political and decision-making circles that could review US policy toward Ukraine.

“Giuliani was not searching for us, we were searching for him,” Telizhenko, the former Ukrainian embassy employee, said of how Ukrainians concerned about political pressure in the country sought out Giuliani.

Giuliani “knew the names, who was connected to who. He’s the filter in all this situation ... He might be the top expert on Ukraine in the United States”.