As the big day dawned last weekend, Berliners held a huge street party, with more than 100,000 revellers gathering in the cold to ooh and aah over an epic display of fireworks. Elsewhere, celebrants of the 30th anniversary of the fall of the Wall followed their own time-honoured traditions. TV producers ran that grainy archive of burly Germans smashing sledgehammers into brickwork, while posh newspapers rolled out their grandest pen-suckers to pronounce on The Future of the Continent. Thus, a historic event that changed the world is stripped of some of its most troubling questions and tidied away into a neat little period piece, a Now That’s What I Call Perestroika! compilation album.

Let us try to change that today, for I come not to bury communism but to praise it – or rather, one aspect of it that gets next to no recognition. On its own terms, “really existing socialism” was a miserable failure: brutally repressive to its own peoples and ultimately unable to compete with capitalist economies. Yet it achieved something else that its own politburos and planners never intended - an achievement that represents one of our era’s greatest paradoxes. Communism didn’t topple capitalism, but kept it honest – and so saved it from itself.

Berlin marks 30 years since fall of the wall in reflective mood Read more

The very presence of a powerful rival ideology frightened capitalists into sharing their returns with workers and the rest of the society, in higher wages, more welfare spending and greater public investment. By sending tanks into Prague in 1968, Leonid Brezhnev may have crushed the dream of “socialism with a human face”; but he and other Soviet general secretaries forced capitalism to become less inhumane. Conversely, the collapse of communism between 1989 and 1991 has left capitalism unchallenged and untempered – and increasingly unviable. The challenge of our time, whether in the UK’s general election or next year’s US presidential contest, is to build a political movement that can restrain a system spinning madly out of control.

You’re not meant to think of it this way. From the very outset, November 1989 was framed as the moment when capitalism could stop battling for survival and finally renew itself. The Guardian’s own leading article, published even as the Trabants queued by the mile to get into the west, summed up liberal hopes: “The wealth is at last available not only to tackle long-neglected evils at home, but to pay for a genuine fight against poverty, injustices and ecological disaster in the rest of the world.”

I trust I’m not giving away too much of the plot by observing it hasn’t panned out like that. Indeed, for most of the past three decades, the political classes – whether Tony Blair or David Cameron, George W Bush or Barack Obama – have accepted vast gulfs between rich and poor as just one of those sad facts of life. With tiny shrugs, they put it down to globalisation or the shifting job market – and deploy the work of top economists such as Harvard’s Lawrence Katz and David Autor who argue that income inequality has been a natural byproduct of technological innovation.

Yet history supplies very different testimony. It shows how the existential threat to capitalism posed by socialist movements, whether revolutionary communist or reformist social democrat, rebalanced power towards workers. The starkest example is the eight-hour working day. Demanded for more than a century by socialists such as Robert Owen, it had been the rallying cry of the annual May Day demonstrations organised since 1890 – yet it took the Bolshevik revolution of October 1917 and the concurrent unrest of workers across Europe to make it law within months in France, Germany and Portugal.

The manner in which socialism reformed capitalism is made explicit in a paper from the International Labour Organisation that notes how the surge in inequality across the west in the 19th century was reversed “as the threat of the spread of communism inspired welfarist redistributive reforms, giving capitalism a more human face”. In 1929 the richest 0.1% of Americans owned almost 25% of all the nation’s household wealth; by the 70s, that had dropped below 10%.

For perhaps a couple of decades after the second world war, the future looked distinctly red-tinted. In 1951, the Canadian news magazine Maclean’s asked: “What hideous magic is possessed by the discredited conspiracy called communism that it continues to control more than a quarter of the voting power in France and Italy?” A few years later, Harold Macmillan’s Conservative ministers were fretting about how the Soviets were outstripping both Americans and the British on education spending, and churning out many more scientists and technologists.

In 1989, capitalism won. Today its greatest ideological challenge is the planet | Larry Elliott Read more

Whether it was the space race or huge European trade unions with links to Moscow, a spectre was haunting the west: the spectre of egalitarianism. In a new study of 16 industrial countries during the cold war, Brazilian economists André Albuquerque Sant’Anna and Leonardo Weller argue that the “more national elites were under the threat of communist revolution, the more the state introduced policies that reduced top income shares” – whether that be taxing and spending more, or allowing uncomfortably powerful trade unions. Those reds under the bed helped ensure you would have social democracy at home.

Communism was not the sole shackle on capitalism – but it is one that typically gets painted out of history. Yet as the late Marxist historian Eric Hobsbawm observed in his memoir Interesting Times, even in the 80s when the Soviet bloc was ailing and feeble, it could still “frighten the rich and the rulers of the world into taking some notice of the needs of the poor”.

Almost all such counterweights to extreme capitalism have disappeared today, from strong trade unions to alternative national models. The resulting debauchery is all around us, from the companies that shovel money towards their shareholders while sending their workers to food banks, to the billionaires now preparing to run for US president, to the major capitals (such as London) that ask barely any questions of passing kleptocrats but instead sell them anything they want.

Countervailing forces made capitalism not only more bearable for those living under it but also more likely to survive. Without them, the system loses both consent and even the will to carry on. The banking crash, Brexit, the impending climate catastrophe – all should serve as distress signals for keen-eared capitalists, yet the answer to each is to serve up more of the same: more finance, more slash-and-burn, more tokenism around carrier bags.

A system so manifestly unwilling to change course will hit the rocks. One paradox of the 20th century was that communism failed, yet helped capitalism to success. The challenge for this century will be whether we can build a new political movement to scare the capitalists silly, so as to force them to come to their senses.

• Aditya Chakrabortty is a Guardian columnist