Great patriots: pro-Moscow protestors wave Russian flags from the Lenin monument in Donetsk city centre, Ukraine, 9 March

With Crimea’s illegal referendum and the peninsula’s annexation by Russia, a new cold war is starting. That does not just mean diplomatic frostiness; it will mean a tense stand-off, sanctions, a military build-up and quite possibly Moscow’s incorporation of further land, including the Russian-speaking regions of eastern Ukraine. At every moment there will lurk the threat of cold war turning into hot war. The Kremlin is well aware how high it has ratcheted up the stakes: state television’s chief propagandist chose referendum night in Crimea to remind the world that Russia is capable of turning America into “radioactive ash”.

The immediate question is how Ukraine – and then the west – reacts to Russia’s takeover of Crimea. Sanctions might hurt, but there is no hope at all that they will force Vladimir Putin to reverse the process. And, short of threatening military retaliation (precisely the thing that could trigger a major war), I cannot see what would deter Russia from responding to manufactured calls from Russian citizens in eastern Ukraine for “help”. On 18 March Putin denied any desire to dismember Ukraine. But he has already authorised the use of force if need be, and between them Ukraine’s far-right nutters and Russia’s provocateurs could easily create the “threat to Russian lives” that would provide the pretext for intervention.

Thus would Europe’s borders be redrawn, and along them a new iron curtain would descend. So much for the hopes we had in those days of revolution from 1989 to 1991, when it seemed we’d all be members of a peaceful, united, de-ideologised continent.

Historians will pore over the origins of this new conflict and see only confusion, lies, misunderstandings and puffed-up egos blundering towards catastrophe. I have long believed that Putin, surrounded by myopic and conspiratorial advisers, does not understand the west, and that the west, so sure of its own righteousness and “victory” in the last cold war, hasn’t even tried to treat Russia with the respect it thought it deserved after throwing off the shackles of communism. Now we are reaping the fruits.

Putin’s “political technologists” have been priming the canvas zealously for the bloody painting being daubed across the continent of Europe. If I were a typical Russian television viewer, with no interest in chasing down alternative reportage, I would be quaking at the thought of what is said to be happening right now in brotherly Ukraine. It’s like the Great Patriotic War all over again; jackboots, brownshirts, swastikas, truncheons; they’re banning the use of Russian; they just showed some millionaire fascist on a stage in the Maidan (Independence Square in Kyiv, the cauldron of the revolution) demanding that Russians be “shot in the head” – and the crowd applauded; “death squads” are being set up, the newsreader said; my sister lives in Donetsk, and my cousin in Kharkov – they’re going to be murdered; and now two people have been shot by fascist thugs . . . you see, it’s starting . . .

Even by the standards of Putin-era television (indeed, even by the standards of Soviet television) the propaganda is jaw-dropping. You have to slap yourself in the face to recall that just a month ago we were watching the opening ceremony of the Sochi winter Olympics – a magical evocation of everything that made Russia great: scientists and writers, composers and cosmonauts, poets and ballet dancers, philosophers and artists. This is the European, cultured Russia we aspire to be, they were saying. Even the Olympic ring that failed to open was somehow endearing, a reminder of what many westerners love about Russia – its maddening foibles, its pretensions to grandeur that often fall just a little short. The producers knew it and made fun of the lapse in the closing ceremony. You see: we Russians can laugh at ourselves. We are just like you.

And then, it turns out, they’re not.

Or are they? Is it we in the west who can’t bear the thought of them being like us? Do we not prefer our stereotypes? Bears, surly Siberians, cold unsmiling Muscovites, gangsters and spies, aggressive communists hell-bent on restoring their evil empire. Much more comfortable. Good to have someone to hate: it makes us feel more virtuous. Did the US secretary of state, John Kerry, who voted for the 2003 invasion of Iraq, not love being able to say to the Russians: “You just don’t in the 21st century behave in 19th-century fashion by invading another country on [a] completely trumped-up pretext”?

Nixon and Brezhnev sign the Strategic Arms Limitation Treaty I in Moscow, 26 May 1972

I sometimes think the west understood the Soviet Union better than it does today’s Russia. For one thing, it was simpler, more black-and-white. But we also had formidable Kremlinologists who knew how to read the signs hidden behind the propaganda. Maybe our foreign ministries today are too obsessed with terrorism and Islam, while Russian studies are dominated (at least in the press and chancelleries, though less so in universities) by experts who, by and large, have a remarkably simplistic view of what is going on. Analysis of Putin’s motives generally amounts to nothing more sophisticated than “he’s a KGB thug, an authoritarian kleptocrat surrounded by corrupt oligarchs, determined to restore the Soviet Union and destroy the west”. Much of that is true! Yet it is only part of the story, and merely describes how he is, but not why, and does not consider whether we inadvertently created a bogeyman.

The Russian view of the west (particularly the one put out for public consumption) is equally flawed, driven by conspiracy theories and mistrust of America’s motives and aspirations. But at least Russia has master diplomats such as the foreign minister, Sergey Lavrov, who knows about the west not from hearsay but because he has studied nothing else for more than 30 years.

When the history of the new cold war comes to be written, its subtitle should be the immortal words of the European Union’s high representative for foreign affairs, Catherine Ashton: “I didn’t pick that up. That’s interesting. Gosh!”

This was her gormless response, in a now notorious leaked phone call, to the Estonian foreign minister, Urmas Paet, when he informed her that opposition gunmen – not President Yanukovych’s snipers – might have been responsible for the mass killings on the Maidan in Kyiv that were the catalyst for the Ukrainian revolution. If this were true, it would be sensational, and have a big impact on the west’s view of the new Ukrainian government.

Yet Paet’s assertion turns out to have been based on a misunderstanding of something possibly said by someone who was in no position to make such a judgement in any case. The minister said he had been given the information by “Olga”, a doctor who had been treating victims. Baroness Gosh had also met Olga, but not been given this incendiary news. Why had they both talked to her? Presumably because the photogenic, English-speaking doctor had appeared on CNN and the BBC, describing the tragedy she was dealing with, and suddenly found herself an important source for top-level diplomats. How often I have seen this happen in my years as a correspondent working in foreign parts: diplomats and journalists swarming around the same little coterie of “sources”, almost always several steps removed from the real decision-making and intelligence.

Poor Olga – Dr Bogomolets – is no forensic scientist, and perhaps something got lost in her (presumably English) conversation with the Estonian. Paet claimed she had said policemen and protesters had been killed by the same snipers: “She can say that it is the same handwriting, the same type of bullets, and it’s really disturbing that now the new coalition, that they don’t want to investigate what exactly happened. There is now stronger and stronger understanding that behind the snipers, it was not Yanukovych, it was somebody from the new coalition.” Dr Bogomolets later denied that she had told him anything of the sort; she hadn’t even seen a shot policeman.

Such was the level of “intelligence” being shared by western leaders as they shuttled in and out of Ukraine, taking decisions apparently way beyond their competence.

Senator John McCain swept in to town and shared a stage with Oleh Tyahnybok, leader of the far-right, anti-Semitic Svoboda party – a man who in many western countries would be a pariah, a politician from the same stable as Jörg Haider, whose election victory in Austria in 1999 caused the EU to impose sanctions against his government. Did McCain know who he was wining and dining with? Did he care? Or is the only qualification for receiving unconditional US support a visceral hatred of Russia?

The US assistant secretary of state Victoria Nuland distributed cookies to the Maidan protesters, and discussed with her ambassador which opposition leader should become prime minister, as though she were viceroy of Ukraine. “I don’t think Klitsh should go into government,” she said. “I think Yats is the guy with the economic experience, the governing experience.”

That would be the former heavyweight boxing champion Vitali Klitschko and Arseniy Yatsenyuk, whose names are rather difficult to pronounce. But she also knew the extremist Tyahnybok, and thought that Yats “needs to be talking to him four times a week, you know”. All this was based on just what knowledge, one wonders. Does this arrogant American have more than the most superficial knowledge of the history and society and needs of the country she is moulding to America’s liking?

The west’s understanding is woeful. How often was the benighted Ukrainian president, Viktor Yanukovych, described as Putin’s great friend or poodle? Like hell he was. The price he extracted from Russia to extend its lease on the Black Sea Fleet base in Crimea in April 2010 (between $40bn and $45bn) made Putin apoplectic. “I would be willing to eat Yanukovych and his prime minister for that sort of money,” he said. “No military base in the world costs that much!” When Yanukovych fled from the Maidan protesters in February this year and turned up in Russia, Putin didn’t even deign to meet him.

How dim must the EU’s foreign policy experts be if they were surprised that Putin trumped their “association agreement” with cheaper gas and a loan of $15bn? The Ukrainian economy is in collapse – of course Yanukovych took the money. The EU spends hundreds of billions to bail out banks, but could not help Ukraine become a democracy.

Our governments appear to be utterly inadequate in foreign policy. Our revolving politicians, one day in education, the next in finance, then at the Foreign Office, may know all about their domestic politics, but abroad (and especially regarding Russia) they are like Columbus setting out to discover India.

Of course getting Russia right is difficult. I count myself pretty well versed in Russian affairs; it’s over 40 years since I started studying the language, the culture, the people, the politics. I have lived there more than ten years in all, under Brezhnev, Gorbachev, Yeltsin and Putin. But I know perfectly well I am an ignoramus compared to Russians. I don’t understand the humour; I could never get the cultural references buried in satirical programmes such as Kukly, the Spitting Image equivalent that Putin banned. It doesn’t stop me pontificating, but knowing how little I understand after all those years, I am horrified to see our flat-footed “diplomats” taking decisions so ill-informed and insensitive that they may be impelling the world towards catastrophe.

But it has always been thus, or at least thus since the collapse of the USSR. No one wants to hear this at a time when Putin is marching his troops into a neighbouring country, but it is perfectly feasible to argue he would not be doing so – that he might not have become Russia’s leader in the first place – if the west had not been so inept in its handling of the collapse of communism.

The first post-Soviet decade – the Yeltsin years – were a disaster for Russia. Americans applauded Boris Yeltsin. He was the kind of Russian we like – “burly” not surly, an iconoclast determined to root out communism, welcoming to western capitalists, comically drunken and impotent to oppose western foreign policies. That the Russian masses were falling into poverty and insecurity was dismissed as a passing phase: it would all come right in the end. That a handful of oligarchs swiped most of the state’s assets and Russia began to resemble a mafia state was no big deal. The oligarchs’ money and TV stations may have been used to rig the re-election of a catastrophically unpopular Yeltsin, but at least it made sure the commies didn’t get back in.

I remember picking my way, as a BBC reporter at the time, through streets full of middle-class people selling off their belongings, to report on Moscow’s first Rolls-Royce dealership. Unsurprisingly, most Russians came to associate capitalism and democracy with financial ruin and humiliation. Some 25 million of them even found themselves outside Russia, living in the new independent former republics of the USSR (not all of which treated their guests with much sensitivity).

The west could have pumped billions into Russia, instead of imagining that freewheeling capitalism was all that was required. It seems our governments had not the faintest idea of how deep the crisis of Russia’s economy was after 70 years of communism, nor of how dangerous the popular mood would become if there was no “cushion”: nothing to save people from poverty, and not even a veneer of respect for the destroyed Russia as a world power.

And that is how we got Putin – brought to power, ironically, by Yeltsin’s own family and advisers. Even they understood that the country needed a jolt. Had Russia not been in such a mess, had “western” policies not been so discredited, the Russians might have chosen a democrat instead.

Putin came to the scene a political ingénu. Today he looks intransigent and single-minded, but at that time he was so inexperienced he opened himself up to all kinds of advice. He surrounded himself with western-oriented, radical reformers. He wooed western leaders, longing to be liked, and mused about joining Nato one day. He offered real help to George W Bush in his war in Afghanistan.

It was just at this point that everything went wrong. Putin was still, at heart, a KGB man, schooled in deception and befuddled by his Soviet vision of the world. He never understood what democracy meant, and began closing down critical media and gathering in power around himself and his quickly appointed clique of KGB comrades. Naturally, the west took fright and began to build up its defences against Russia – even though, at this point, Putin had shown no ill intentions towards other countries whatsoever.

George W Bush’s understanding of Russia was, I guess, about as good as his understanding of Iraq: international affairs reduced to a few soundbites. Ignoring Russia’s protestations, he pressed on with a missile shield, allegedly to defend against Iranian rockets but in fact positioned in such a way that the Kremlin saw its own strategic defences weakened.

What the point of this was, God only knows. The system doesn’t work anyway (it’s like trying to hit a bullet with a bullet, from hundreds of miles away) and in any case Iran has since all but given up its nuclear arms pretensions.

Russia desperately wanted to be part of Europe’s security architecture, but Nato expanded eastwards towards Russia’s frontiers, thus making Russia, ironically, more of a threat than it would otherwise have been. In return, the Russians started building up their own defences. In 2008 Nato promised eventual membership to Ukraine, exactly what Putin now fears will happen as the country turns westwards.

Yet what if the west, instead, had calculated that Putin could have been persuaded to rein in his authoritarian tendencies in exchange for proper clout in world affairs? Cleverer diplomats might have persuaded him. The result could have been the kind of Russia we wanted – democratic, peaceful, not threatening . . . and therefore a welcome asset at the global table.

By encircling Russia and undermining its security (which Nato expansion and the missile shield undoubtedly did), we created the enemy we didn’t want. Halfway through his second term, Putin decided that America did not want to share power in the world. And he was right – not with a man who was locking up his critics and rigging elections. Both sides were sliding into a spiral of mutual mistrust and hatred. For Putin, the battle for acceptance was lost and it was no longer worth “improving” himself to regain it. He became the menacing, vengeful warlord we now have to deal with. Gosh!

Bill Clinton’s old Russia hand Strobe Talbott describes the upheaval in Ukraine today as Putin’s payback to the west, particularly the United States, for what he “sees as a quarter-century of disrespect, humiliation and diplomatic bullying”. In his speech on 18 March, Putin resentfully listed all the grievances that have built up over the years, concluding that the centuries-old policy of “containing Russia” continues.

To be clear, what Putin has done in annexing part of Ukraine is unacceptable and should be punished, though goodness knows how this can be achieved without precipitating war. We can probably never have a sane relationship with Russia until Putin and his henchmen are gone.

Yet perhaps, one day, the Russia we saw at the Sochi Olympics opening ceremony will be not just a figment of the imagination, but something we can all celebrate and welcome into our hearts. However, our next generation of western Kremlinologists should bear this in mind: whoever is in the Kremlin – even the most likeable, “western” leader you can imagine – will have Russia’s interests at heart, not ours. They will want a say in the world commensurate with Russia’s size and nuclear status, they will care about Russians living abroad, and they will resist anything they see as a threat to their security.

Angus Roxburgh is a former BBC Moscow correspondent. His book “The Strongman: Vladimir Putin and the Struggle for Russia” is available in an updated paperback version (I B Tauris, £12.99)