My first memory of Vladimir Putin – if you can call it a memory – goes back to late 1991, just a month before the collapse of the Soviet Union, when I caught sight of him, without knowing who he was, of course, in St. Petersburg. I was making a series of reports for the BBC in the city, which had just been given its original name back, after 67 years as "Leningrad." As we filmed a meeting between the mayor, Anatoly Sobchak, and a visiting British politician, a small, fair-haired man flitted anonymously in the background.

Rewatching the footage 20 years later, I recognize the features: soft, thin hair parted to one side; glassy eyes; and protruding lips. He walks with his head pressed forward and an aggressive gait, rolling slightly from side to side. This is Mr. Putin at 39, recently returned from a five-year posting as a spy in East Germany and now head of the city's "external relations committee." He is unobtrusive and slightly nervous, just as you would expect from a man used to living in the shadows. He fingers his chin self-consciously, knowing a Western TV camera is pointed at him – possibly for the first time in his life.

Putin's job was to attract foreign investors to the city. He would later succeed in bringing in giants such as The Coca-Cola Company. But in 1991 his immediate priority was to solve the city's food crisis – a colossal task, as I saw for myself when I toured the St. Petersburg "food depot."

In the communist system, all agricultural produce was brought to an enormous central area, to be sorted and transported to the city's shops. Don't imagine a Western-style fruit distribution center, where apples and oranges are individually wrapped in tissue and packed into shock-resistant boxes, then whisked out to retail stores. In a system devoid of incentives, almost all the produce went to waste. Workers were fishing through crates of potatoes that had already turned into a stinking black mush, picking out the few that could be salvaged and tossing them into another crate. Eventually a few of them might have reached a shop, and some might even have been sold and eaten.

Putin's task was to arrange emergency food supplies from the West. It was a job that, I think, had two profound effects on the future Russian leader and may still shape who he is today as he's about to assume the presidency for another six years as one of the country's most enduring, enigmatic, and controversial rulers in modern history.

First, it placed him at the center of the most humiliating moment Russia had endured for perhaps half a century. His country was on the brink of starvation. The Soviet planning system had collapsed, and the half-baked reforms introduced by Mikhail Gorbachev had merely introduced an element of chaos. Long lines of angry customers were forming outside empty food stores. Russia was forced to beg for humanitarian aid, like some third-world country. A fellow correspondent in Moscow once coined the phrase "Upper Volta with rockets." That was really how the decaying Soviet Union looked, and Putin knew it. When he came to power nine years later, he vowed never to let that happen again.

But being in charge of St. Petersburg's food imports affected Putin in another way, too. He was a novice in politics and business. His boyhood had been spent dreaming of joining the KGB, and his entire career had passed in the secret world of surveillance and espionage. For five years he had been based in Dresden, East Germany, watching Moscow's sway over Eastern Europe slowly crumble. On his return to Russia in 1990 he had stumbled into politics almost by chance. And not just into politics, but into the glittering world of foreign trade and investment – a world that soon presented him with great temptations.

Since the city had no hard currency with which to buy food imports, he was given the task of arranging barter deals – supplying Soviet oil and other raw materials in exchange for imports of foodstuffs. In 1992 an investigation by members of the St. Petersburg City Council established that the raw materials had been duly exported, but the food supplies – to the tune of $92 million – had never materialized. Responsibility for the missing largess fell directly on Putin's shoulders, and the council demanded his dismissal. His boss, Mayor Sobchak, stood by him, but the whiff of corruption has stuck to Putin ever since.

When he later became president, Putin presided over the creation of a state in which corruption is so widespread and so complex that one Russian businessman told me I would never, as a Westerner, understand it: "Theft is not theft as you know it. It is the entire system – the political system, the business establishment, the police, the judiciary, the government, from top to bottom, all intertwined and inseparable."

Putin surrounded himself with cronies from his previous life – from the St. Petersburg administration, from the KGB, even friends from his judo club, and co-owners of a "dacha co-operative" – a settlement of private country houses on a lake outside the city. He gave them the best positions in government, and allowed them to commandeer the most lucrative sectors of Russia's economy, its banks and mass media. One of the leading opposition figures, Alexei Navalny, has dubbed Putin's party, United Russia, "the party of crooks and thieves" – a coinage so successful it almost certainly contributed to the party's poor performance in last December's parliamentary elections.

•••

Back in the 1990s, Putin's rise to the top was precipitous – and it remains something of a mystery how a little-known, middle-ranking intelligence officer, tainted with allegations of corruption, could have achieved such a meteoric career. In St. Petersburg he had cloaked himself in the kind of democratic credentials that were vital for advancement in the Boris Yeltsin period, becoming head of the pro-government Our Home Is Russia party in the city.

He was also skilled at making allies, who helped him move from St. Petersburg to Moscow. Here he swiftly climbed through the Kremlin ranks, spending less than a year in each position: deputy chief of staff to the president, director of the FSB (successor to the KGB), head of the Security Council, prime minister.

Putin appears to have impressed President Yeltsin's family and closest advisers not only with his vitality and can-do attitude, but above all with his strong sense of loyalty toward his benefactors. When his former boss Sobchak became embroiled in a corruption scandal, Putin stuck by him and helped him escape to Paris to avoid prosecution. Putin also showed great loyalty toward Pavel Borodin, a Russian official who gave him his first job in the Kremlin. When the state prosecutor started investigating Mr. Borodin for involvement in a multimillion-dollar bribery and money-laundering scam, Putin – now FSB chief – got rid of the prosecutor by using a well-worn KGB trick: secretly shot footage allegedly showing him with prostitutes. (Putin later put up $3 million in bail to get Borodin out of a Swiss jail, where he was serving time for the kickback scheme, which involved a Swiss construction firm.) His new patrons evidently understood he would show the same loyalty to Yeltsin and his family – and that his secret service connections were a help, not a hindrance.

He finally stepped into Yeltsin's shoes when the president resigned on the last day of the 20th century. Ten days earlier Putin had told a gathering of secret service agents: "I want to report that a group of FSB operatives, sent to work undercover in the government, is successfully carrying out its mission." Maybe it wasn't entirely a joke.

Putin, though now acting president, was still scarcely a public figure, and when he allowed a cameraman to follow him on one of his first days in the Kremlin he looked gauche and unsure of himself. In unbroadcast footage he appears weirdly detached from the world around him. He hasn't bothered to look and see what view there is from the window of his new Kremlin office and is surprised when he pulls back the curtains. His desk is empty, apart from a couple of papers, one of which he quickly turns face down because it is from the FSB. At home, in the presidential residence, he is taken aback when asked about the furniture because he simply hasn't thought about it: For this austere man, home comforts apparently mean nothing.

It didn't take long for the secretive secret agent to transform himself into the man we know today – confident, brusque, abrasive, given to coarse language. A Russian journalist whom he met over lunch when he was still FSB chief describes his ability to mimic his interlocutors, to empathize with them and make them feel comfortable. Soon the new president was able to make foreign leaders feel at ease. George W. Bush famously looked into Putin's eyes and felt he got a sense of his soul. For his part, Putin soaked in and copied the manners and self-confidence of the world leaders he now mingled with.

The Putin we have gotten used to over the past 12 years is a strange mixture: On the surface we see the global politician, smartly dressed, brilliantly well-informed, and quick-witted; but under the veneer we also sense the ghost of the Leningrad school brat, the youth who, by his own admission, readily lost his temper and got into scraps. "I was a hooligan," he told interviewers shortly before he was elected president in 2000.

It was the coarse Putin of the Leningrad backyards we heard when he told a French journalist who dared to question him about the ferocious bombing of Chechnya: "If you're such a Muslim sympathizer, come to Moscow. We can have you circumcised!" It was the slumdog Putin who threatened to hunt down terrorists and wipe them out – to have them "scraped from the bottom of the sewers."

It was the unsophisticated Putin, his view of Western democracy conditioned by years of Soviet propaganda and KGB training, who said it was "normal" for demonstrators in the West to be "beaten about the head" by police, and who once told President Bush that the United States wasn't a democracy because the president was elected "not by the people but by an electoral college." "Vladimir," Bush whispered to him, "don't say that in public – it'll only show you don't understand our system at all."

It was a Putin conditioned by decades of suspicion about Western plots who also told Bush he "knew," because his secret services had informed him, that America had special factories producing substandard poultry exclusively for export to Russia.

•••

Between 2006 and 2009, I observed Putin and his entourage at close quarters, working as a media adviser to his press office. (My task, in which I admit I failed, was to educate them about Western media practices and persuade them to adopt a more open style of government.) In the past three years, in the course of making a series of television documentaries and writing a book about Putin, I interviewed dozens of politicians in Russia and the West who had dealings with him. From these experiences I have detected several factors that I believe are crucial to his personality and behavior.

Regarding his attitude toward the West, it's important to remember that Putin was not always the iron man, the obstructionist we see aiding the Bashar al-Assad regime in Syria or ranting about Western "interference" in Russian affairs. What we see here is the anger of a man who feels spurned – and he is not a man to forgive easily.

In his first years as president, Putin went to great lengths to be accepted by the West. He asked NATO's secretary-general, "When are you going to invite Russia to join?" He moved swiftly to forge friendships with the leaders of Britain, the US, France, and Germany – but found that only the latter two treated him the way he wanted. He made overtures, such as offering the US unprecedented help in prosecuting its war in Afghanistan (albeit for the selfish reason that he hoped this would help him fight terrorists inside Russia). He invited world leaders to lavish celebrations in St. Petersburg and Moscow. He introduced economic reforms at home that genuinely impressed the West.

But Putin felt he got nothing in return for all these efforts, and genuinely could not understand why the West – while paying lip service to a new "friendship with Russia" and "the end of the cold war" – routinely ignored Russia's security interests. Despite developing what appears to have been a sincere, if superficial, friendship with Bush, Putin had to watch as NATO took in East European states, expanding right up to Russia's borders, and America abandoned the Antiballistic Missile Treaty and pressed on with plans to build a missile defense shield in Europe. The antimissile system is ostensibly aimed against a potential future threat from Iran, but Moscow is quite convinced – and not entirely without reason – that it could be used against Russia, too, and that at the very least it destabilizes the balance of power that has kept the peace for decades.

On top of these strategic issues, Putin was furious that the US refused to abolish the antiquated Jackson-Vanik amendment, which restricted trade with Russia, and – in Putin's view – kept moving the goalposts for Russia's accession to the World Trade Organization. All these grievances accumulated like bad blood, until, in February 2007, he exploded in a landmark anti-American speech in Munich, Germany.

The US Defense secretary, Robert Gates, sat in the front row, feeling the full force of Putin's invective. Mr. Gates was not entirely unsympathetic to Putin's gasp of despair, and during talks in Moscow later that year, made some unprecedented concessions to the Russians, offering them a 24/7 presence at the missile defense installations being planned for deployment in Poland and the Czech Republic.

Gates felt this would assuage the Russians' fears that the system might be aimed against them. The Russians were astonished by the offer – it was the kind of gesture that could have kick-started a whole new relationship, by making the Russians feel included in the West's defense plans rather than threatened by them.

But it came to nothing. Gates had been winging it. When he took the idea back to Washington, it was immediately shot down by the defense establishment. "When we got the offer in writing," Russian Foreign Minister Sergei Lavrov told me with a derisive smile, "not one of the proposals was in it."

Instead of improving ties, incidents like that made things infinitely worse. Putin began to feel he could not trust anything the Americans said. The distrust was mutual. In my role as an adviser to the Kremlin, I tried to explain why it was that the West did not trust Russia. Don't you see, I would say, that if you are cracking down on democracy at home, if you are taking control of all the media, if you refuse to condemn the Soviet past – and even treat Stalin as if he had been just a normal leader – then people in the West are bound to look at you with fear?

The answer was always the same: The West shouldn't lecture us about democracy. We will do things our way.

•••

This brings me to the crux of the problem – the point at which foreign relations and internal politics intersect. Putin is a complex and, in many ways, misguided character. His understanding of "democracy" sees nothing incongruous about the state controlling the media or police beating up demonstrators. But grafted on to this KGB-inspired, controlling mind-set is something the West rarely appreciates – Putin's fear that the West is actively meddling and is determined to "destroy" Russia.

When I first heard him using such language, I took it as mere rhetoric. Now I think he really believes it. He believes the conspiracy began with the "Rose Revolution" in Georgia in 2003 and the "Orange Revolution" in Ukraine in 2004, which brought pro-Western leaders to power. Because American-funded pro-democracy groups played a prominent role in these revolutions, Putin believes they were entirely sponsored by the West and would not have happened without Western interference. He believes the recent demonstrations in Russia have the same backing and the same aim – to overthrow his regime.

In fact, both the US and Russia tried to influence the Ukrainian election of 2004. One of Putin's spin doctors, Sergei Markov, sent to Ukraine to help the pro-Russian candidate, spoke to me in apocalyptic terms about what he believed the West was up to: "These people were determined that Ukrainians and Russians should start killing each other – and I mean killing each other." Another spin doctor, Gleb Pavlovsky, said it was part of a "Destroy Russia" project.

The disturbing thing is not that Mr. Markov's and Mr. Pavlovsky's words are utterly fanciful. It is that Putin almost certainly thinks the same way.

If Putin really believes the protests against him are all orchestrated by the West, it could be his fatal mistake. When he and President Dmitry Medvedev announced last September that they intended to swap roles – with Putin standing for election and appointing Mr. Medvedev as his prime minister if he won – they explained that their reasoning was that Putin's poll ratings were higher than Medvedev's. What they both failed to understand was that their arrogance in simply arranging to "castle" (the chess term the Russians use for the move) was the very thing that would trigger the collapse in Putin's ratings.

First the previously invincible Putin found himself booed at a martial arts event. Then in December, after parliamentary elections were blatantly rigged to achieve victory for his United Russia party, thousands poured into the streets to protest – for the first time since Putin came to power.

Later that month he showed himself to be out of touch with what was going on. In a televised question-and-answer session, he mocked the protesters' white ribbons as looking like condoms, and claimed the demonstrators had been paid by Western agents. He described them as "Bandar-log" – the name of the "monkey-folk" in Kipling's "The Jungle Book."

It is worth analyzing this comment, because it was no unrehearsed line. The Bandar-log are not just monkeys: Kipling describes them as undisciplined, leaderless, chattering, full of fine ideas but unable to carry anything through to a conclusion – exactly how Putin describes the opposition. He remarked, "I have loved Kipling since I was a boy." (In fact, like most Russians, he probably knows his Kipling better from a series of Soviet animated cartoons made in the late 1960s.)

In his television appearance, Putin referred to a Bandar-log scene in the book that is quite frightening. The monkeys are rioting, and only the giant python, Kaa, is able to calm them – by mesmerizing them and calling on them to step closer ... so he can consume them for his supper. Putin paraphrased Kaa's words, with a wry smile on his lips: "Come to me, Bandar-log!"

It would seem that Putin really believes he has the rioting "monkeys" fully under his control. If so, it could be a fatal error. Moscow's "chattering classes" are convinced that Russia's political scene has changed dramatically. Already, an anti-Putin candidate has been elected mayor in a provincial city. The opposition may be disoriented, having found its voice only a few months ago after years of enforced silence. But it is not about to bow its head before Kaa.

Now, speculation is rife about where Putin will go from here. Will he make compromises with the resurgent opposition, to remove its sting, or will his undemocratic instincts hold sway? For the West, one thing is clear: It will probably have to deal with the prickly Putin for another six years, and it must decide how to make the best of that. In my view it would be pointless to lose those years in cold-war-style confrontation. Putin's foreign policy has always been reactive. He responds to positive gestures with goodwill, and to pressure by pulling down the shutters or even lashing out.

Get the Monitor Stories you care about delivered to your inbox. By signing up, you agree to our Privacy Policy

So perhaps it is time to tempt him with another Gates-style gesture on missile defense – but this time, meaning it. If, as we have seen, he is suspicious of the West's intentions, then maybe it is time to reassure him. Maybe, just maybe, Putin will respond – he might become more cooperative in dealing with Syria and Iran, and if he feels more secure he might even be persuaded to loosen up at home. The alternative would be six years of cold-war standoff, which would benefit neither the West nor the democrats inside Russia who are hoping for change.

• Angus Roxburgh, a former longtime Moscow correspondent who once worked for Putin, is the author of "The Strongman, Vladimir Putin and the Struggle for Russia."