On a Friday night last November Vladimir Putin was running three hours late. A group of foreign academics, journalists and selected Russian TV cameras were quarantined in a restaurant in an equestrian centre. Deadlines were lapsing and Putin's guests began asking questions about the odd location. Everything from the oak beams, log fires and snug bars had been rigged. The venue had been constructed for this one meal.

Putin finally emerged wearing a ski jacket. He stopped short in the entrance with his hands down but away from his sides. An unseen hand removed the puffer jacket, another slipped an elegant sports coat on to his shoulders. Putin hardly paused, but in a flash he had changed roles. He emerged on the other side of this catwalk as the tanned chief executive of Russia Inc.

He had just made the biggest mistake of his career. In front of millions of people, he forced his protege, Dmitry Medvedev, to nominate him for another two terms as president. In a spectacular miscalculation of political timing, Putin had destroyed not only Medvedev's fledgling career as a reformer but severely damaged his own. He had made nonsense of the elections that followed – a parliamentary vote in December and the presidential one this coming Sunday – because everyone already knew the result. This might have worked for the old Russia – passive, fatalistic, offline – but the new, pushy middle class was not buying it.

Four mass demonstrations later, Putin's campaign is on a knife edge. He has to be elected president on the first round on Sunday. If he succeeds, most political analysts in Russia are agreed that a third term of office as president will be a transitional one. There is unlikely to be a fourth.

This was not in the script. The man who had pulled Russia back from the brink of collapse had a vision. That vision – to restore Russia to what he saw as its rightful place in the world, one which was not just deserved but ordained – needed him to stay in power for at least another two terms. Not anyone else, just him, Vladimir Putin. How did the man with the penchant for Soviet slang and the swagger of the new Russia, lose his greatest asset: his political nose for what Russians thought?

Bleak beginnings

Putin's family lived through one of the bleakest periods of Soviet history, the Nazi siege of Leningrad. His father was a commando who fought behind the lines, and was one of only four to return from one mission. The family lost their first son before the second world war, and their second to diphtheria during the war; when Putin was born in 1952, he was regarded as miracle of fate.

Putin's childhood in a communalka – a communal flat – was tough. The boy was weedy, an outsider and bullied. He dreamed of turning himself into a street fighter and became, in his own words, a real hooligan. Bewitched by Soviet films, he tried to join the KGB, but it turned him away. The KGB needed graduates with law degrees, not street hoods. Putin responded by studying international law at Leningrad State University, and joined the KGB on graduation.

A star spy he wasn't. He worked briefly in counter-intelligence, monitoring foreigners in Leningrad, before being dispatched as a case officer to Dresden. According to a German account, Putin's mission was to recruit functionaries from the German Communist party and the Stasi to back Gorbachev against the anti-reform East German leader Erich Honecker.

But the empire was all too quickly collapsing around him. He returned to Leningrad – now St Petersburg – and went to work for one of his old law lecturers, the city's mayor, Anatoly Sobchak.

Gleb Pavlovsky, a former Kremlin adviser who was ousted last year after lobbying for Medvedev to stay as president, knows the Russian leader well.

"Putin belongs to a very extensive but politically invisible layer of people who after the end of the 1980s were looking for a 'revanche' in connection with the fall of the Soviet Union," Pavlovsky said. "I was also one of them. My friends couldn't accept what had happened. There were thousands of people like that in the elite, who were not communists – I was never a member of the communist party. They were people who just didn't like how things had been done in 1991. By revanche I mean the resurrection of the great state. Not a totalitarian one, of course, but a state that could be respected. And the state of the 1990s was impossible to respect."

Until 1996, Putin was a virtual unknown. But he had also become indispensable to the democrats because of his skills in taming the monster that the early Russian democrats had created through their chaotic privatisations of the Russian economy: the oligarch.

"Putin comes in with a double-edged offer to these oligarchs," said Clifford Gaddy, senior fellow at the Brookings Institution thinktank. "On the one hand he would protect them against each other. He was the only guy who could ensure that infighting did not explode and cause a total destruction of all the oligarchs and bring down the whole system. And on the other hand, he could protect them against all the demands, populist demands, demands by the communists and others that these guys be strung up from the nearest lamp-post."

The way he achieved both was to get his hands on a monopoly. Not of oil or gas, but of a far more important commodity in post-Soviet Russia: information. Putin had headed the central control directorate for one of his early mentors, Yeltsin-era 'reformer' Anatoly Chubais, and made sure all other potential rivals for the collection of financial information were purged. This was the suitcase full of secrets Putin needed. It did not just contain the oligarchs' original sins – where the skeletons were hidden, the corruption, the bribes – but also the financial information that any corporation, even a squeaky clean one, would spend millions of pounds to protect.

The astronomical sums of money involved may have been new to Putin but the method he used to dominate this small group of commercial princelings was not.

Putin's skill as a KGB case officer was to "work with people". What that meant was to work on people. Gaddy said: "Putin understood the principles of the British intelligence chief John Masterman's double-cross system: don't destroy your enemies. Manipulate them and use them for your own goals."

This was exactly the man liberal economists like Chubais and Alexei Kudrin thought they needed when they brought him into the Kremlin in August 1996. And it worked, for a time. Putin gave almost all of the oligarchs the protection, or in Russian criminal jargon "the roof" that they craved. When he asked something from them, like funding for some dinosaur of the defence industry, their duty was to step up to the plate. And to stay out of politics.

"Putin does not believe he can purchase the loyalty of people in his whole apparatus, because there is always the possibility that someone comes along and offers something more," Gaddy said. "Putin figured out that the only way he could control the oligarchs to guarantee their loyalty, or at least ensure they did not act against him, was to have some threat hanging over them, something he knew about their activity that would ruin them and their businesses if it were to be exposed. It's the definition of blackmail. And that is a more powerful weapon or lever against people than simply deriving income."

What Putin created was not a restoration of the Soviet state, although he undoubtedly dreamed of it. Nor was it totalitarian, because Putin's reach was not total. He created instead a personalised state, one with his name written right through it like a stick of rock.

Fiona Hill, director of the Centre on the United States and Europe at the Brookings Institution pointed out that in 1996 Boris Yeltsin had complained that Russian history had known monarchism, totalitarianism, perestroika and finally democratic development. Each period, Yeltsin said had its own ideology, but contemporary Russia had none. Hill said: "What Putin gave us in 2000 was Putinism. What made it different from what we had before was that it had someone's name attached. Totalitarianism was sometimes called Stalinism but not always. Perestroika was never called Gorbachevism. But Putinism was invented as the answer to Yeltsin's question. And a central idea for the state can not be that. It has to be beyond one person. That's the history of Russia, the quest for a national idea that stands beyond one person."

Putin lost no time in creating his own cult following. As Boris Yeltsin was swaying under the influence of drink and a series of mini-strokes, Putin, his prime minister, was active – and foul-mouthed.

The man who promised to follow Chechen separatist fighters "into the shithouse" and kill them there did just that, with brutal consequences, in the second Chechen war. As president, he appeared barechested on horseback, at the controls of a Tupolev bomber, on the conning tower of a submarine. The Russian public had, it seemed, an inexhaustible thirst for Putin the action man. Pavlovsky said it had a "Viagra" effect on his ratings. Until the day it didn't.

In November last year Putin came to a martial arts fight in Moscow. He climbed into the ring with a microphone to congratulate the winner Fedor Emelianenko and started: "Dear friends, today all lovers of martial arts and ultimate fighting have a great celebration …" before his speech was drowned in a chorus of boos.

Warnings ignored

There had been warning signs. Mikhail Dmitriyev, a pollster who uses focus groups to monitor what Putin's supporters are saying about him, detected a steady deterioration of his political brand as far back as 2006. Students had started demonstrating and disparate political groups – nationalists, communists, liberals, groups that in the past Putin had been able to buy off – began finding common cause.

"There is a universal demand which underpins the personal dissatisfaction with Putin as a leader and which unites both pro-communists and the market-minded middle class, and that is the sense of the unfairness," says Dmitriyev, president of Centre for Strategic Research.

"And this is not just about corruption. Traditionalists, the pro-communist pensioners' group, dislike the behaviour of the elite with its conspicuous consumption achieved through illegally-acquired wealth. The elite talks about public welfare but in practice they steal and spend. This is considered complete hypocrisy.

"The urban middle class, on the other hand, says the system is unfair not because of the unequal income distribution but because of the selectivity of justice. They want equal treatment under the law. You can't go through red traffic lights with your blue flashing light on your car, just because you are in power," Dmitriyev said.

Andrei Soldatov, an investigative journalist, also felt something stirring. He had already published the English language version of his latest book on the Federal Security Service (FSB) called the New Nobility, and expected the Russian version to disappear down a black hole of domestic disinterest. It didn't. It began selling like hotcakes.

If sociologists like Dmitriyev were warning something was going wrong with the image of the governing elite, Putin's famed security services should have been drawing the same conclusions and doing something about it. But they weren't either. In 2010, when the tectonic plates of Russian opinion began to shift the FSB was strangely passive.

"An open conflict was raging inside its ranks," Soldatov said. It was not been between services or individuals, but generational. The FSB generals, all men Putin personally knew and appointed, were in their early sixties and rapidly approaching retirement age, which, under Russian law can only be prolonged by the personal decision of the president. With villas to hang on to in Rublyovka – the playground of the rich outside Moscow – the personal fate of the generals became dependent on that of their head of state.

By the end of 2010 they were convinced that Medvedev would get a second term. A special Medvedev re-election unit was set up inside the FSO, the Federal Protective Service which guards the Kremlin. Medvedev himself toyed skilfully with the retirement trap in the FSB. There was even a rumour circulating that Medvedev had prepared a list of up to a dozen generals who would be fired in the near future. The very existence of such a list froze the top FSB brass likes rabbits in the headlights.

Underneath them, the colonels fumed, their path upwards to the lucrative reaches of the top brass well and truly blocked. There are two consequences for a security service with an internal conflict, Soldatov said. "The first is that the channel for disseminating information is broken. Nobody talks to each other and if the colonels have good information about what is going on in opposition circles, they are not interested in passing it on to their generals, and if they did, the generals for their part would not trust it. The second is that under such a system it is difficult to form groups to support one particular general. Without support and resources, the FSB loses its control."

If the FSB, Putin's own service, was that dysfunctional, it is safe to assume other parts of his tightly drawn circles of advisers were affected by the same paralysis. Was it just the passing of time - 12 years in power - that had made Putin rusty? Or was there something predestined about the collapse of his image as the strongman of Russia?

Back to the future

There are two questions Putin faces in this third and final period of office. Can he separate himself from the dead hand of the elite around him – can he separate himself from Putinism? The second is even harder to achieve. Can Russia separate itself peacefully from him?

Putin, Pavlovsky said, is not a cynic: "He thinks man is a sinful being and it is pointless to try to improve him. He believes the Bolsheviks who tried to create fair people were simply idiots, and we wasted a lot money and energy on it … So Putin's model is that you need to be bigger and better capitalists than the western capitalists. And more consolidated as a state: there must be a full, maximum unity of state and business."

Sir Andrew Wood, former British ambassador to Moscow, agreed. "He prizes and gives loyalty to the gang, but is unforgiving to his enemies," he said. " 'The weak get beaten' is a constant theme. Rules for him are more for other people to obey than ones which bind the powerful. He has a powerful memory and ability to intuit others' needs and hopes, but in a suspicious way. He has no great attachment to truth if untruth suits his purposes. He is no stranger to violence. These qualities seem to me to be constants."

Putin by Pavlovsky's account is no great believer in independent political parties: "Putin doesn't believe that there is real competition between political parties in the west. He thinks of it as a game, a kind of golf in a private club: someone is stronger, someone is weaker, but that in fact there is no real competition. Putin imagines it like that. He imagines it as in the Federal Republic of Germany after the war under the leadership of Konrad Adenauer. So there are two parties, one of which has power, and the second waits, it may wait a long time. And then, Putin always said, at some time in the future the opposition will receive power, and when it does we must be ready for that moment. And by being ready, I mean we (the Kremlin) must be in both parties.''

For Pavlovsky, United Russia's collapse in the polls in December was obvious. "It was nothing more than a telephone system for conveying the Kremlin's orders. It transmitted a signal. It has absolutely not independence and can not act on its own. It needs full instructions – 1,2,3,4,5. If 3 and 4 are missing, it stops and waits for instructions."

The masses, in Putin's mind, cannot be trusted to rule. Pavlovsky said: "What is the Putin consensus? It is a construction of a pact between the ruling elite and the main groups of society in which society is guaranteed a degree of social distribution and an elite which feels free to move across borders. This state has no ideology. It will be a state without ideas oriented on common sense, the average man, the citizen. Nonetheless the masses must not be given access to power."

The Putin consensus began when Putin decided he was the sole bearer of that guarantee and he controlled the whole situation. But it ended at precisely point he decided to return to power as president: "That was his mistake. His decision to return was a certain delusion of grandeur, " Pavlovsky said.

Professor Richard Sakwa at Kent University agrees that Putin's system of governance is deeply flawed and that elections had become plebiscitary, ratifying decisions already made outside the electoral process. But he believes that Putin can change if he heeds the right advice. Sakwa is impressed with a former Kremlin insider like Sergei Mironov, the head of Just Russia, a party that the Kremlin created but has gone on to take its first gulps of air on its own.

"Mironov showed acute awareness of the dangers arising from the stalemated political situation in the country. He is a classic case of the former insider going into opposition. Just Russia represents yet another attempt to give expression to social democratic ideas in Russia, of the sort long advocated by Mikhail Khodorkovsky and Mikhail Gorbachev", Sakwa said.

Mironov, in an interview, himself was cautious about the change going on in Russia; "Putin doesn't want fundamental change, and neither in fact do most Russians. Even those who came out on Bolotnaya Square and Sakharov Avenue and those offended and humiliated by the cheating of the 4 December election. Every day they experience corruption and completely ineffective work of the state authorities and they understand that something needs to be done. But even they do not want fundamental, sudden changes. They are for evolution. Putin has positioned himself as the person who can ensure stability, but his stability looks like stagnation to me."

But the rest of Russia inhabits a different land. "My focus groups from provincial cities now indicate that the person the people want as president is the year 2000 version of Putin. So, if Putin comes out as being the same passionate reformer as he was then people are ready to vote for him. So if Putin could win in the first round, it can't be ruled out," Mironov added.

If Putin does re-invent himself, it will be only by taking on as prime minister one of the economists who first brought him to Moscow in the first place – Alexei Kudrin. Key liberal advisers to Medvedev like Igor Yurgens have already jumped ship and gone over to Kudrin's camp .

Fiona Hill said: "My best bet is that you will see more and more people being brought to the government and eventually taking it over. If Putin does not start doing that, he will risk a massive upheaval and all bets are off, because there is no system to take its place. What he did in 2000 was significant in terms of restoring the central state, but those means have come to an end. It all about regime survival for him now … Putin has forgotten about Russia, and the people on the streets are saying 'We are Russia too, but we are not part of your system'."

Putin faces a Hobson's choice: to select effective managers capable of running the show called Russia Inc is very dangerous for him personally, because they will by definition not be part of the team – his charmed circle from which no-one can escape. But the dangers for Putin and for Russia of staying with same time-serving crew of telephone engineers, the mere transmitters of signals, are even greater. Russia could end up, by splitting up altogether, as it threatened to twice in 1991 and 1999. That is still a real fear in Russian minds.

Sir Andrew Wood said: "The stagnation issue is present to the minds of the group. The fact that Putin goes out of his way to deny it is indicative. He and they have spoken of a bright future, but bringing in fresh blood if by that you mean fresh ideas as well would threaten group cohesion, and perhaps Putin's ability to arbitrate. If you mean replacing some of the lesser figures, then that would be a different matter. Nothing that has happened so far either in response to the protests or in what Putin has said in the course of the Duma or presidential campaigns indicates that he sees his return to the Kremlin as meaning the sort of renewal that would be the result of an election campaign elsewhere. The great themes are continuity and stability."

Putin left the restaurant that night in November the same way he came in. He stopped at the door and his favourite ski jacket reappeared on his shoulders. But he had lost his cool several times in the previous three hours, when the same question popped up: How does Russia move on without you? His answer was: "Don't bury me yet."

• This article was amended on 28 February 2012. The original should have named Rublyovka, not Dubrovka, as the playground of the rich outside Moscow. Owing to an editing error, the piece said that Vladimir Putin's parents lost their first son to diphtheria; that was actually their second son.

Pocket Profile



Born Leningrad, USSR, 7 Oct 1952

Career to date Spy, government fixer, prime minister, president. Chiefly known for reviving the Russian economy, the cold war and the spy movie genre. Otherwise to be found undertaking vigorous outdoor pursuits, often barechested and always bearing a passing resemblance to the former England cricketer Nasser Hussein

High point Winning things such as the presidential elections and wars with small, downtrodden Caucasus peoples

Low point Being booed at a wrestling match in November 2011, the moment at which it became apparent that a tsar was falling

He says "We'll follow terrorists everywhere. Should we catch them in a shithouse, we'll whack them in a shithouse."

They say "When I looked into Putin's eyes, I saw KGB." (Colin Powell)

"Medvedev continues to play Robin to Putin's Batman." (US ambassador John Beyrle)