The current panic about Russia is a curious phenomenon. By any objective standard Russians are freer in the authoritarian state established by Putin than at any time in the Soviet Union. Many are also materially better off. Russia has abandoned global expansionism, and is now a diminished version of what it has been throughout most of its history - a Eurasian empire whose chief concern is protection from external threats. Yet western attitudes are more hostile than they were during much of the cold war, when many on the left viewed the Soviet Union, which was responsible for tens of millions of deaths, as an essentially benign regime.

To see how this state of affairs has come about one must understand the progressive narrative - embraced nowadays as much on the right as the left - that shapes western perceptions. The Soviet collapse was a defeat for communism, a prototypical progressive ideology. There was never any prospect of post-communist Russia embracing neoliberalism, another western model. Something like Putin's Russia was always on the cards, but the return of history isn't part of the progressive script. Most of our leaders are disciples of Woodrow Wilson, with a religious faith in what Francis Fukuyama only the other day described as "the march of history towards global democracy". Prosperity brings bourgeoisification and liberal values, or so they believe. Russia - rich, nationalist and authoritarian - doesn't fit this progressive fairytale, and the west's reaction is a mix of threatening bluster and mounting panic.

Nothing is more misguided than talk of a new cold war. What we are seeing is the end of the post cold war era, and a renewal of geopolitical conflicts of the sort that occurred during the late 19th century. Their minds befogged by fashionable nonsense about globalisation, western leaders believe liberal democracy is spreading unstoppably. The reality is continuing political diversity. Republics, empires, liberal and illiberal democracies, and a wide variety of authoritarian regimes will be with us for the foreseeable future. Globalisation is nothing more than the industrialisation of the planet, and increasing resource nationalism is an integral part of the process. (So is accelerating climate change, but that's another story.) As industrialisation spreads, countries that control natural resources use these resources to advance their strategic objectives. In deploying energy as a weapon Russia is not resisting globalisation but exploiting its contradictions.

We are back to great-power politics, shifting alliances and spheres of influence. The difference is that the west is no longer in charge. With their different histories and sometimes sharply conflicting interests, Russia, China, India and the Gulf states are not going to form any kind of bloc. But it is these countries that are shaping world development at the start of the 21st century. The US - its bankrupt mortgage institutions nationalised and its gigantic war machine effectively funded by foreign borrowing - is in steep decline. With its financial system in the worst mess since the 1930s, the west's ability to shape events is dwindling by the day. Sermonising about "law-based international relations" is laughable after Iraq, and at bottom not much more than nostalgia for a vanished hegemony.

Deluded about its true place in the world, the west underestimates the risks of intervening in Russia's near abroad. Russia's weaknesses - demographic decline, cronyism in the economy and a seething sense of national humiliation - are well known, but western vulnerabilities are no less real. Our leaders bore on about Russia needing us as much as we need Russia. In fact, despite a recent blip, investment in Russia is a byproduct of the global market that will continue for as long as it continues to be profitable, whereas Russian energy supplies can be curtailed at will by the Russian government. Economists will tell you the country is too reliant on oil. But the world's oil reserves are peaking while globalisation continues to advance, and Russia stands to gain from any international conflict in which supplies are disrupted. Again, the west needs Russia if the Iranian nuclear crisis is ever to be defused peacefully, and without Russian logistical cooperation Nato forces will find it even harder to bring the aimless, unwinnable war in Afghanistan to any kind of conclusion.

Right-thinking bien-pensants in all parties believe Russia would be more amenable to western interests if only it were more truly democratic. But Putin is wildly popular precisely because he is asserting Russian power against the west; if he were more accountable to public opinion he might be harder to deal with. Democracy has numerous advantages, but it is no guarantee of a reasonable foreign policy. The current Georgian imbroglio is itself a spin-off from democratic politics. Mikheil Saakashvili's reckless incursion into South Ossetia, where Russian forces had been stationed under international agreements for 16 years, was most likely encouraged by elements in the Bush administration in the hope of damaging Obama in the run-up to the presidential election. The gambit may have worked, but the result has been a conflict that increases Russia's leverage over the flow of oil in the region and strengthens Iran in central Asia. If Dick Cheney's pledge of support for Georgia during his travels last week was a move in the Great Game it was spectacularly ill judged.

Clearly, with the exception of some in "old Europe", our leaders do not know what they are doing. The grandstanding of David Miliband and David Cameron in Ukraine illustrates the point. Blathering about national self-determination and territorial integrity, they seem not to have noticed that the two principles are normally incompatible. Self-determination means secession and the break-up of states. In the Caucasus, a region of multi-sided national enmities, it means a wider war and worsening ethnic cleansing. The stakes are even higher in Ukraine. Deeply divided and with a major Russian naval base in the Crimean port of Sevastopol, the new state will surely be torn apart if an attempt is made to wrench it from Russia's sphere of influence. The country would become a battlefield, with the great powers irresistibly drawn in. Playing with Wilsonian notions of self-determination in these conditions is courting disaster.

Let there be no mistake: Russia is, in some respects, a dangerous state. With their background in the security services, its leaders are ruthless pragmatists who will use any means to achieve their objectives. Their goal may be to roll back western influence in Russia's near abroad, but their strategy is to take whatever they can. Perceiving the west to be in decline, they are testing whether it has any coherent strategy to protect its interests. From what we have heard from our leaders, it does not.

A start would be to shelve plans for further Nato expansion, while making it unequivocally clear that existing commitments in eastern Europe and the Baltic states will be honoured. At the same time every effort must be made to reduce Europe's dependency on Russian energy. Western leaders need to acquire a capacity for realistic thinking, or else they will be woken from their dream of progress by the force of events.

· John Gray is emeritus professor of European thought at the LSE.

comment@theguardian.com