Working on one of my final stories as the Guardian and Observer’s Moscow correspondent, I took my seat at the Bolshoi Theatre one evening last December for the premiere of Nureyev. The ballet told the story of superstar dancer Rudolf Nureyev, who defected from the Soviet Union to the west. The premiere had been postponed at the last minute some months previously, partly due to its gay themes. In the interim between the cancelled premiere and the rescheduled one, the ballet’s director, Kirill Serebrennikov, known for his risque productions and political activism, had been placed under house arrest on embezzlement charges that most people thought were spurious. He’s still there and risks a long jail sentence.

Inside the Bolshoi’s gilded, celebrated auditorium, where tsars held their coronation concerts and Lenin and Stalin gave speeches from the stage, the hall was packed with the elite of Vladimir Putin’s Moscow. Theatre and television stars mingled with government officials and Putin cronies and they all sat through the powerful staging that dealt with the search for personal and artistic freedom under the oppressive Soviet regime.

It did not require great leaps of thought to transpose the layers of history on to the present day. At the end of the performance, the cast came on stage wearing T-shirts and demanded that Serebrennikov be freed. They were applauded from the stalls by the representatives of the regime that had locked him away. The inspiring but unnerving evening was a distillation of all the things that had first drawn me to Moscow. The coexistence of beauty and horror, hope and despair, glory and absurdity is frustrating yet alluring. Russia gets under your skin.

For me, it’s been a particularly long journey. I first set foot on Russian soil as an 18-year-old in January 2000, a few weeks after Putin had first been made acting president by Boris Yeltsin. I leave, all these years later, with Putin about to be re-elected for another term. I’ll be back for the World Cup, but after that I’m off for real. Honestly.

A large part of why the region is such a repository of extraordinary stories is the overarching context of the collapse of the Soviet Union. More than a quarter of a century later, the aftershocks can still be felt strongly. Social, political and economic change all came at extraordinary speed back in 1991 and for most people it was not for the better. Every single story I’ve written from Russia and the other former Soviet countries over the past years has been, at least in part, about the Soviet collapse. Reformatting national ideologies, recalibrating the international geopolitical balance and restoring the psychological makeup of more than 250 million people cannot happen overnight.

I saw the consequences of the Soviet collapse vividly on my first trip to Russia in 2000. Life in the decade since 1991 had progressed along the lines of a particularly implausible episode of a job-swap reality TV show: biochemists were now taxi drivers, market stallholders were CEOs. The criminals became the authorities and those who tried to stand up against them became the criminals. A few people had pilfered all the ladders, leaving the rest to be devoured by snakes.

I spent four months teaching English at a secondary school in Moscow, then several weeks traversing Russia in the third-class platskart carriages of Trans-Siberian trains. These dormitories on wheels puttered across the endless Eurasian landmass, the air inside thick with a blend of sweaty feet, fish lunches and a sooty tang emanating from the coal-fired samovars that dispensed hot water for tea. I remember a palpable sense of confusion with the state of affairs in the new country. A few young business-oriented types saw it as a time of great excitement and opportunity, but most people seemed lost on some kind of existential level – plaintive, overwhelmed and alarmed by the chaos that a decade of “democracy” had brought. Two years earlier, a financial meltdown had meant that millions of Russians lost whatever paltry savings they had managed to put aside.

Public opinion surveys from the time show that the majority of people were unimpressed with the new Russia. At the end of 2000, 75% of people said they regretted that the Soviet Union had collapsed. “Most of the population didn’t recognise the Russian Federation as a real thing,” I was told much later by Gleb Pavlovsky, an adviser to both Yeltsin and Putin. “They felt like they lived in some kind of strange offshoot of the Soviet Union. We had to ensure the handover, but we also had to create some sense of nation.”

This, in the broadest terms, was Putin’s mission.

At the end of 2003, I returned to Moscow after studying Russian and Soviet history at university. I worked for an NGO for a year before taking up journalism. The city was slowly becoming more prosperous. Over the next decade, oil prices rose so high that, even allowing for the rampant corruption in Putin’s inner circle, money did trickle down and provide real benefits to people in the cities. In Moscow and other major settlements, abject squalor was disappearing from the central streets and a middle class began to develop. With it came coffee shops, wine bars and frequent flights to Europe.

Many of my Moscow friends seemed to live life at a faster, more exciting pace than people I knew in London. It was not unusual to be speaking to someone who ran their own company, had at least one marriage and two children under their belt, and find out that they had not yet turned 30. Unsurprisingly, given the turbulence of the past decades, people tended to live in the moment and nobody thought about savings or pensions. Moscow life was turbulent, unpredictable and extremely fun.

The journalism, too, could be fun. In what other region of the world would a top official mention as an aside during an interview that he had been abducted by aliens in a yellow spaceship? Where but in the former Soviet lands would a journalist get to meet a reclusive billionaire who had for years lived incognito in a glass castle among a collection of exotic pets, before deciding he wanted to become prime minister, and winning? (Those were the leader of the Russian region of Kalmykia and the leader of Georgia, respectively.) Over the years, I’ve met oligarchs and warlords, shamans and terrorists, mad scientists and musical prodigies. Of course, every corner of the globe is interesting, but I can’t help thinking that the former Soviet countries do this combination of depressing, inspiring and bizarre like nowhere else.

Not all the reporting was fun and there was plenty to be depressed about in the suffocating crackdowns on civil society, the egregious inequality and the horrendous social problems. Millions of Russians drank ethanol, window cleaner, perfume or other industrial spirits that were marketed more cheaply than vodka. Grave heroin and HIV epidemics were worsened by a refusal to back any treatment except abstinence. When authorities cracked down on heroin supply, people switched to krokodil, a synthetic opioid made from cooking codeine pills, lighter fluid and industrial cleaning products into a brownish gunk. In the city of Tver, I met krokodil addicts who had patches of rotting flesh on their bodies, where they had injected the drug. “Look at this,” said one of them, halfheartedly flinging an arm to signify the miserable urban sprawl around him. “Wouldn’t you also want to shoot up?” There was both an economic and an existential sense of hopelessness, interconnected and mutually reinforcing.

Putin’s goal has been to imbue this creaking nation with a new vitality and create a sense of national unity. For his first decade in power, the gradual economic improvements were enough to satisfy many Russians that things were gradually going to get better. Hard ideology was thin on the ground and the Kremlin spin doctors (or “political technologists” as they are called in Russia) who employed it often did so cynically.

Russians had watched the once-rigid ideological constraints of the Soviet Union crumble and had then seen the lofty democratic slogans of the 1990s disintegrate in an orgy of stealing. It led to a situation where nobody really believed in anything at all. The Kremlin manipulated the political playing field, creating pocket opposition parties, liberals or nationalists, and removing them again if they grew too popular. I once visited a political technologist who was working on an electoral campaign for a regional mayor who had been using “traditional values” rhetoric. He welcomed me to an office covered from floor to ceiling with Orthodox religious icons; behind his desk was an extra-large image of a dusky Byzantine Christ. I mentioned to him that the devout backdrop seemed slightly at odds with his outfit, a black jumpsuit and a fluorescent orange bandana. “Oh, I’m not at all religious,” he told me with a laugh. “I just like to change my ideological surroundings every few weeks for inspiration.”

‘One of the only narratives to rouse genuine feelings was the Soviet victory in the Second World War.’ Photograph: Nikolai Sitnikov/TASS via Getty Images

One of the only narratives to rouse genuine feelings was the Soviet victory in the Second World War, or the Great Patriotic War, as it’s still known. After a 1990s consumed with soul-searching about the darker sides of Russia’s past, Putin wanted Russians to take pride in their history again. The book I’ve written about my time in Russia, The Long Hangover, is partly about Putin’s attempts to overcome the legacy of the Soviet collapse and restore a sense of pride to Russians, particularly through the war victory. Putin appealed to many historical events and differing ideas of Russianness, but almost organically, the war established itself as the foundational stone of the new nation. The victory of 1945 was the answer to the trauma of 1991. “Through you, we got used to being winners,” he told veterans in 2000. “This entered our blood. It was not just responsible for military victories, but will also help our generation in peaceful times, help us to build a strong and flourishing country.”

For a wounded nation that had few victories to celebrate in living memory, the war was a powerful rallying point. Almost every family had some connection to the war, in which the Soviet sacrifice was unimaginably huge, but gradually under Putin, the darker sides of the war effort and the Stalin regime that ran the country at the time were pushed to one side. The understandable search for national pride gave way to jingoistic chest-thumping. War rhetoric was also used to colour the present-day narrative; again, Russia alone faced a rapacious enemy – this time, the US. Of course, all countries have a selective approach to their histories and the German ambassador to the UK recently suggested that the Brexit vote at least partially came about because Britain can’t get over its obsession with the Second World War. But in Russia, the selective memory reached truly disturbing levels and the glorious war narrative became something akin to a civil religion, with its own saints, martyrs and unimpeachable truths.

The logical conclusion of this increasing bellicosity came in 2014. The Maidan revolution was answered by Russia’s annexation of Crimea and then the months of war in eastern Ukraine. Thousands were killed, including the passengers of flight MH17, almost certainly shot down by a Russian missile meant for a Ukrainian military plane. There were many reasons why Putin decided to annex Crimea and get involved in eastern Ukraine, but among them was the clash between two nations still attempting to create their new national identities after the Soviet collapse. The second half of my book is about the events of 2014 and its pages are filled with those who struggled to find new identities in the post-Soviet world.

One was Igor Bezler, a wild-eyed, pro-Russian commander known as the Demon. A former Soviet and Russian special forces officer, in 2014, he and his heavily armed men took over the town of Gorlovka and meted out summary justice to its residents. When I visited him at the height of the war, the interview turned sour and he threatened to execute me. I fled quickly when he changed his mind, but a year later I met up with Vasil Budik, a man whom the Demon had held hostage for months.

By then, Budik was working for the Ukrainian defence ministry, having been released in a prisoner exchange. He said I had had a miraculous escape. Just before my visit, the Demon had executed a rebel fighter for rape. Budik said some of the things he knew the Demon had done were so awful that he would only talk about them to a war crimes tribunal. But he also, extraordinarily, said he was still in touch with the Demon and the pair had spoken on Skype the previous week. He told me that the Demon had many qualities of a good officer and, when I speculated to him that he might be suffering from Stockholm Syndrome, told me that I shouldn’t break people down into such simplistic categories.

“The Demon gave an oath to the Soviet Union. An oath is something a man should give only once in his life. For all that he fought for Russia, I think he remained loyal to the Soviet Union. You fight for your motherland and then your motherland disappears. Not everyone can handle that.”

After 2014, anti-western sentiment in Russia rose and reporting on the Kremlin became more difficult. Putin’s inner circle shrank further and some of the few doors that had been ajar to me were slammed shut. The idea that the west was working for regime change in Russia, and that the foreign media were merely an arm of this policy, became more widespread. At one meeting with a reasonably high-placed official, whom I’d hoped to cultivate as a source, my interlocutor pounced on me:

“You think we’re barbarians, don’t you?”

“Of course not! That’s a ridiculous thing to say,” I said. He fixed his eyes on me: “Well, we are barbarians, OK? But it’s your fault. Why can’t you just leave us alone? Why can’t the west just stop interfering in our affairs? You did it in 1917, you did it in 1991, and now you are again trying to bring the country to a collapse and you will only succeed in making us more angry! We know things are wrong here, but we will fix them ourselves! Leave us alone!”

Alexei Navalny ‘scores low ratings in the polls, but he is a charismatic orator and his message about official corruption in the inner circle is powerful.’ Photograph: Pavel Golovkin/AP

The paranoid outburst was an insight into the curious duality so often present in the Russian psyche: aggression and insecurity. I often think that Russians and Americans have much more in common than either might like to think, but the way their patriotism is expressed is one big difference. If flag-waving American jingoism is uninterested in how the world views America, Russian patriotism is fixated on the west and often rooted in an inferiority complex. This is not that surprising given what Moscow lost when the Soviet Union collapsed and Russia became a geopolitical irrelevance. Much of Putin’s time in office has been about trying to restore this influence. Sometimes, he has succeeded: the intervention in Syria has reshaped the Middle East and forced the west to have Russia at the table, even if Russian casualties could become a problem for the Kremlin. At other times, he has overreached: the interference in the US election may have won Donald Trump a few votes and Putin some notoriety, but it has also made Russia a toxic subject across the western world. Robert Mueller’s indictments of 13 Russians for election interference this week are one more sign that the scandal is not likely to abate soon.

As the international political situation deteriorated over recent years, the quality of life in Moscow kept improving. Part of the response to protests against Putin in 2011 and 2012, largely driven by urban elites, was a beautification project designed to make Moscow nicer and distract people from politics. This process continued after 2014. Moscow became a city where it was pleasant to take long walks, where excellent coffee shops abounded and where service staff and their customers were no longer gratuitously rude to each other. In my 20s, there had been smoky dive-bars that stayed open until 7am and decadent parties; now, there were classy wine bars, landscaped parks and restaurants that I actually wanted to eat in.

Whether the plan to divert attention has worked is harder to gauge. For now, political apathy reigns both among the urban elite and the broader population and Putin is sure to win next month’s election. In a remarkable sleight of hand, Putin manages to portray himself as part of the solution rather than part of the problem, even though he has been in charge for 18 years. While his approval ratings are real, much of the support is fragile and predicated on there being no alternative. On my trips around the Russian provinces, people often told me they were unhappy with their lives and despairing of corrupt officials. Kremlin messaging has long been that the alternative to Putin is revolution and chaos and has done everything to make sure that this is the case, strangling any opposition movements that might grow naturally before they can even take off. Remembering the economic hardships of the 1990s that followed the last government collapse, many Russians prefer to stick with Putin.

The opposition politician Alexei Navalny has emerged as the biggest potential threat. He is never featured on state television, he and his team deal with constant harassment, and his brother has been jailed. On trips to the Siberian hinterlands, I’ve met young, enthusiastic Navalny supporters who are slowly winning around their parents and friends to the cause. Navalny scores low ratings in the polls, but he is a wily politician and a charismatic orator and his message about official corruption in the inner circle is powerful if it gets wide exposure. Putin is on firm ground batting away challengers who want more liberal politics or rapprochement with the west; he has few answers if confronted with hard proof of the obscene wealth of his ministers and his childhood friends.

Putin will win another six-year term on 18 March, but there is no clear ideological message for his next term and nobody knows how – or if – he will relinquish power in 2024. It is hard to imagine Putin simply stepping down, but also unclear for how much longer he can keep control over his courtiers, who are engaged in vicious feuds behind the scenes for money and influence. One high-placed source, who once upon a time would lecture me that I should look at the long arc of history, that countries with the historical baggage of Russia do not democratise overnight and that Putin is moving Russia in the right direction, looked rather demoralised when I last saw him. He admitted that nobody knew what was coming next and that the medium-term prognosis was rocky. He simply shrugged and said: “Countries go through good patches and difficult patches. This is a difficult one.”

It will be strange for me to follow what happens next in Russia from outside its borders. For generations, Russians have either by choice or political exigencies fled their homeland only to find that nowhere else quite fills the Russia-shaped hole in their soul. Émigré memoirs are filled with a sense of toska, the untranslatable Russian word that Vladimir Nabokov called “a sick pining, a vague restlessness, mental throes, yearning”.

Foreigners with the Russia bug are not immune to toska either. So while I’m ready to leave, I suspect that one day I’ll be back. Having seen nothing but Putin’s Russia in the 18-year span of my time there, I’m fascinated by what post-Putin Russia will look like, whether we have to wait six or 26 years to find out.

Shaun Walker has been Moscow correspondent of the Guardian and Observer since 2013. From March, he will become Central and Eastern Europe correspondent, based in Budapest. His book, The Long Hangover: Putin’s New Russia and the Ghosts of the Past, has just been published.