The past is a foreign country; they do things differently there. We can see just how alien the past can be by taking my time machine for a short spin back 20 years. For many readers, especially younger ones, time tourism will be a shock. In 1998, Amazon is a company struggling to convince people that there is a profitable future selling books online. Facebook doesn’t exist. Neither does the iPhone. The Russian intelligence service is run by Vladimir Putin. Some things haven’t changed then.

Also in 1998 – and this will really surprise some people – Tony Blair is the most popular prime minister Britain has ever had. He and other centre-leftists of his type are dominant in the western democracies. Bill Clinton, a “new” Democrat, is in his second term at the White House. “New Labour” has recently surged to power with a parliamentary landslide in Britain. It will go on to win two further elections. The “neue mitte” – the word new is much loved by this generation of social democrats – has been a winner for the SPD’s Gerhard Schröder, who is embarked on the first of two stints as Germany’s chancellor. The moderate left is in government in two-thirds of the countries that are members of the European Union. Their successful offer is broad support for free markets combined with good public services, a decent welfare state, internationalism and social liberalism. This seems to be a magic formula both for the taking of power and the exercising of it. At around this time, one of Mr Blair’s senior advisers told me that they represented “a new common sense” so potent that neither the traditional conservative right nor the old socialist left could hope to compete with it.

How archaic that sounds when we return to 2018. Just about everywhere you look, social democrats are being pulverised. The latest example has been furnished by the populist earthquake in Italy where Matteo Renzi’s Partito Democratico was smashed down to less than a fifth of the vote and the centre-left came in third behind a rightwing bloc fronted by Silvio Berlusconi, who is banned from taking public office, and the Five Star Movement, which was founded by a man who is, literally, a comedian. An even more dismal fate befell the French socialists when their candidate for president finished fifth with less than 7% of the vote. They then went on to lose 250 of their 280 seats in the national assembly. When Germany went to the polls last autumn, the SPD, for decades the most powerful centre-left party in Europe, recorded its worst result since the creation of the federal republic in 1949. Even though its junior role in the previous “grand coalition” with Angela Merkel was electoral hemlock, the SPD has gone back into another one for fear that a fresh election would produce an even more dire result.

It is true that centre-right parties have also been haemorrhaging support to the various insurgent brands of illiberal populists, demagogic nationalists and fascists. The troubles of the centre-right are scant consolation for the centre-left because its crisis looks much more existential. Social democrats neither head the government nor lead the opposition in Germany, Britain, France or Italy – Europe’s four largest economies. The centre-left was thrown out of power when Austria went to the polls and fell to a historical low in the most recent Spanish contest. In Scandinavia, traditional heartland of European social democracy, they are in charge in just one country. The Czech social democrats and the Dutch Labour party have been marmalised by the voters. There are no social democrats at all in the Polish parliament. Portugal’s social democratic government stands out because it is such a rarity. Britain has exhibited this trend in its own eccentric way. The social democrats who used to control the Labour party have been replaced by people hailing from much further left along the spectrum. Jeremy Corbyn’s leadership cadre has an ideology sufficiently to the taste of the Communist Party of Great Britain that it says it won’t run candidates against Labour at our next election.

There is one thing to be said for the bleak place in which social democrats find themselves. They have time to reflect on what went wrong. They were too often managerial and metropolitan with the result that they lost connection with segments of traditional support that felt condescended to by a cosmopolitan elite. After the scarring electoral defeats at the hands of the right in the Thatcher-Reagan era, the centre-left overcorrected in its approach towards markets. They were too indulgent of the excesses of high finance in the run-up to the Great Crash of 2008 and have been duly punished since. They were too mesmerised by the power of globalisation and the dividends to be had from it; they paid too little attention to those who lost out or felt left behind. Throw in stagnant incomes for many folk combined with conspicuous, and sometimes obscene, gains for those lucky enough to be rich in assets. Stir in surges in immigration and it has been a perfect storm.

Luigi Di Maio, left, the Italian Five-Star Movement’s leader, with its founder Beppe Grillo. Photograph: Alessandro Di Meo/EPA

Some argue that it is even grimmer. Social democracy is not just in distress – it is defunct. Hegemonic in 1998, it has become essentially obsolete in 2018. It is contended that the “third way” of the Blair era was only viable in “good times”. The formula depended on strong and stable economic growth to satisfy the public desire to enjoy better services and welfare provision without paying too much more in tax. That formula doesn’t work when growth is anaemic, money is tight and choices are much more stark.

Another gloomy view is that social democracy is the victim of a realignment that is replacing the traditional division between left and right with a politics more driven by identity and values. A split between “open” and “closed” views of the world is polarising population groups and opinion between nativist authoritarians and globalist liberals. This is agony for social democrats because it cleaves their historical voting coalition of the working class and middle-class liberals. Understanding their plight has been easier than finding ways to put themselves back in contention. One approach is to try to woo voters away from populists by echoing elements of their messages. Sweden’s ruling social democrats, who face an election this year, are fighting to stay in power with a strategy entitled “better welfare, law and order, and faster integration”. The centre-left in Denmark is taking this experiment further. The Danish social democrats have made a hard swing to the right on immigration in an attempt to stop blue-collar voters going over to nationalists. If-you-can’t-beat-the-populists-join-them is a strategy freighted with a lot of risks.

It is a sign of desperate times for the centre-left that this is happening in Scandinavia, once the bastion of tolerant and open European social democracy. In Britain, Labour has just about managed to straddle the open/closed divide, but it helps to be in opposition where you can fudge the harder choices. Labour will find it much tougher going if it should find itself in government.

There is a paradox about this crisis for social democracy. The broad formula of the centre-left still has appeal to many millions of voters. There is little evidence that the modern electorate wants to embrace the heavy-metal socialism of a super-statist society. Nor can we detect a great clamour to live in a state-shrunk society of unfettered markets and devil-take-the-hindmost. Many voters may have given up on social democrats, but they still like the idea of a regulated market economy with good public services and decent welfare protection. This is the western Europe that social democrats were so influential in creating in the decades after 1945. Progressive taxation, equality of opportunity and the idea that the strong have a responsibility towards the weak are basic tenets of the centre-left that have become embedded. It was so successful that it could make social democrats of Conservatives as when Britain’s Tories accepted the Labour-created NHS. Witness also the rather social democratic way in which Theresa May’s government is proposing interventions in the energy and housing markets.

Social democrats have arguably had more influence over the development of western Europe than any rival political movement. They created a world that electorates have, by and large, come to take for granted. This may be part of their problem. Fewer and fewer voters are old enough to have memories of how ghastly some of the alternatives can be. When electorates get experience of what the snake oil peddled by the populists really tastes like, social democrats may be given an opportunity to be heard again. They had better be ready with attractive things to say and compelling leaders to articulate them.

• Andrew Rawnsley is an Observer columnist