1962. At the airport, Harry Palmer – not yet played by Michael Caine, not in fact even named in Len Deighton's original novel – stocks up on his reading. For the flight he buys the New Statesman and History Today. And then he adds a copy of the Daily Worker. Not just because our Harry (as we might as well call him) is a British spy, keeping up with the communist enemy, but also because Harry, unlike the uppercrust nitwits he works for, is classless and intelligent and up-to-the-minute, and so in a menacing way at this moment in the 20th century does communism seem to be, thanks to the public image of its homeland the USSR.

For Harry, knowing about the Soviet Union is a way of keeping the sad old, creaky old, shabby-genteel world of England ironically in its place. He's fighting it, but its existence is an asset to a grammar-school oik on the rise, like him. Out beyond the bedsits and the stale crumpets and the golf-club ties, there's a giant waking, and it's proof positive that the old order of things is shiftable, that there can be novelty under the sun. Two years later in the sequel to The Ipcress File, an eggyolk-stained has-been explains pityingly to Harry that there's no way a low-rent place like England is ever going to induce a Soviet scientist to defect. "Simitsa works with refrigerated ultra-centrifuges. They cost around £10,000 each. He has 12 of them." That same year, 1964, the classless and up-to-the-minute Harold Wilson makes it part of his pitch to the electorate that the sad old, creaky old British economy should be supercharged with some Soviet-style scientific efficiency. And the voters buy it, white heat, "National Plan" and all.

This was the Soviet moment. It lasted from the launch of Sputnik in 1957 through Yuri Gagarin's first spaceflight in 1961 and dissipated along with the fear in the couple of years following the Cuban missile crisis in 1962. (It was already going, in fact, at the time of the 1964 election; it was a piece of Wilson's appeal that was premised on a fading public perception and was dropped from Labour rhetoric shortly thereafter, leaving not much behind but a paranoid suspicion of Wilson among egg-stained, old-school-tie spooks.) But while it lasted the USSR had a reputation that is now almost impossible to recapture.

It was not the revolutionary country people were thinking of, all red flags and fiery speechmaking, pictured through the iconography of Eisenstein movies; not the Stalinesque Soviet Union of mass mobilisation and mass terror and austere totalitarian fervour. This was, all of a sudden, a frowning but managerial kind of a place, a civil and technological kind of a place, all labs and skyscrapers, which was doing the same kind of things as the west but threatened – while the moment lasted – to be doing them better. American colleges worried that they weren't turning out engineers in the USSR's amazing numbers. Bouts of anguished soul-searching filled the op-ed pages of European and American newspapers, as columnists asked how a free society could hope to match the steely strategic determination of the prospering, successful Soviet Union. President Kennedy's aide Arthur Schlesinger wrote a White House memo sounding the alarm over "the all-out Soviet commitment to cybernetics". While the Soviet moment lasted, it looked like somewhere which was incubating a rival version of modern life: one which had to be reckoned with, learned from, in case it really did outpace the west, and leave the lands of capitalism stumbling along behind.

Which didn't happen. Which didn't happen so thoroughly that the way the Soviet Union seemed to be between 1957 and 1964 or thereabouts has been more or less displaced from our collective memory. In the quick, associative slideshow that assembles itself in our heads these days when the USSR (1917-91) is mentioned, the bits with the flags and with Stalin's moustache now lead on directly to the images of the country's dotage, when old men in ugly suits presided over an empire of antiquated tractor factories before Gorbachev came along and accidentally put the whole thing out of its misery. The era when the place seemed to be in a state of confident, challenging, expansive maturity has fallen off our mental carousel. If in the 70s the USSR turned out to be only "Upper Volta with rockets" – in the words of an American diplomat unimpressed with the way the metalled roads ran out only a few kilometres outside Moscow – then it must have always, and only, been Upper Volta with rockets.

The idea of an enviable Soviet Union utterly fails to compute. We tend to assume therefore that the Soviet moment must have been pure illusion. Perhaps a projection of western fears; perhaps a misunderstanding of what the headline feats such as Sputnik implied about the rest of Soviet life. It had been a reasonable assumption, for nervous western onlookers in the early 60s, that a society which launched satellites must also have solved simple everyday problems such as supplying lettuces and children's shoes. When it turned out that it wasn't so, that the Hemel Hempstead branch of Start-Rite would have represented unimaginable luxury in a Soviet city, the space rockets stopped signifying a general, enviable "high technology". They started looking like some pharaoh's pet project, a pyramid scraped together on the back of poverty, cruel and a bit ridiculous.

But the image of the USSR that the west briefly nurtured in the late 50s and early 60s was not a pure illusion. It was an exaggeration of something real; a report of a real confidence, a real feeling of success in Moscow which the west helped falsify by translating it into western terms and tricking it out with the west's expectations. Something really did go right or go well, then, for the Soviet Union, which we're in danger now of tidying away, like all episodes in history that point in a direction not taken and which therefore refuse to fit into the hindsighted narrative we make out of the past for our convenience. The truths learned later about the Soviet economy were quite real, of course. It did indeed prove to be wasteful rather than efficient, cack-handed instead of strategic, alarmingly incoherent rather than terrifyingly rational. But if we tell ourselves only a case-closed story of communism as an inevitable disaster we miss other parts of the past's reality and foreclose on the other stories it can tell us.

Give your imagination permission to engage with some unlikely facts: in the 1950s, the USSR was one of the growth stars of the planetary economy, second only to Japan in the speed with which it was hauling itself up from the wreckage of the war years. And this is on the basis not of the official Soviet figures of the time, or even of the CIA's anxious recalculations of them, but of the figures arrived at after the Soviet Union's fall by sceptical historians with access to the archives. The Soviet economy grew through the second half of the 50s at 5%, 6%, 7% a year. As Paul Krugman has mischievously pointed out, the USSR's growth record in the 50s elicited exactly the same awed commentary as Chinese and Indian growth does today. Admittedly, "growth" did not mean exactly the same thing in the Soviet context that it did in, say, the American one (average for the period 3.3% a year) or in the British one (average: 1.9%; have a stale crumpet). Soviet growth was counted differently, was biased massively towards heavy industry and did not necessarily imply a matching growth in living standards.

Yet there had been a palpable transformation in the way Soviet citizens lived. In 1950, as in 1940 and 1930, they had been wearing hand-me-downs and living for the most part in squalid, crowded "communal flats" carved out of antiquated pre-revolutionary buildings. In 1950, you could be director of a major Moscow hospital and live behind a curtain in 1/17th of a Tsarist ballroom. Ten years later, Soviet citizens were wearing new clothes and moving in ever-increasing numbers into new apartments with private bathrooms; they owned radios and pianos and were beginning to own fridges and televisions too. In 1960, the hospital director would be sitting pretty in a sunny new-build out in the Sparrow Hills and driving to work in a well-waxed sedan with the leaping-stag logo of the Gaz company gleaming on its bonnet. Going by the measure of the capitalism of the 30s, which is what the Soviet Union had first set out to beat in terms of living standards, Soviet life was now spectacularly prosperous. The USSR could now feed, dress, house and educate its people better than depression America or Nazi Germany. If capitalism had remained unchanged, the Soviet Union would at this point have looked like a reasonable, if tyrannous and polluted, version of the earthly paradise.

Mission accomplished, materially speaking. Instead, of course, capitalism had unfairly shifted the target by doing some growing of its own. Which was why, even on a generous estimate, the average Soviet income still only amounted to 25% or so of the average American one; not bad at all, compared with the recent Soviet past, and positively inspiring from the point of view of (to pick two Soviet allies) India and China, yet not really economic victory. But the Soviet march to wealth was not finished. This was only the halfway stage on the road to a far greater abundance.

According to Marxist theory, the USSR had been on a long, strange detour ever since the October revolution. Marx had predicted that communism would come in the most advanced of the capitalist countries, not in backward, roadless, shoeless, illiterate Russia. He had supposed that the plenty of the socialist future would be built on top of all of the cruel-but-necessary development work of capitalism – that socialists would inherit a machine they only had to perfect and to direct towards the satisfaction of everybody's needs, rather than the needs of a few top-hatted owners. The Russian situation was utterly different, and so the Bolsheviks had been obliged to operate a socialism which was doing capitalism's job for it. They'd bootstrapped an industrial base out of virtually nothing, to produce the steel and cement and machine tools on which any further advance depended. They'd trained a workforce and disciplined it in the rhythms of industrial life. They'd educated a peasant society till it was bristling with science degrees. They'd also killed several million people, and massively out-brutalised the capitalist version of the industrial revolution, all in the name of humanity; but their information was limited, thanks to the paranoically limited bandwidth of the channel through which they viewed the outer world, and the vision of capitalism with which they compared their own record was Marx and Engels's portrait of Manchester a century earlier as a laissez-faire heart of darkness. They could point out to themselves that while they had the smokestacks and the squalor and the cruelty and the black grime on every surface, they also had palaces of culture offering ballroom-dancing lessons and opera at low, low prices.

In any case, the job was now done, and history could resume its rightful course. Atop the steel and cement could grow the pastel pagoda of utopia; Marx's utopia, that deliberately underdescribed idyll where wonderful machines purred away in the background, allowing the human beings in the foreground to "hunt in the morning, fish in the afternoon, rear cattle in the evening, criticise after dinner, just as I have a mind . . ." So rich and comprehensive would be the flow from the mechanical horns of plenty that it wouldn't even be necessary to measure out the goods in proportion to the work people did. Everyone could have anything and be anything. If you've ever read one of Iain M Banks's "Culture" novels you'll recognise the setting, except that this post-scarcity paradise was to be run on the advanced technology of the mid-20th century, rather than the science of a galaxy far, far away; spun up from artificial fibres and pneumatic mail and computers made of glowing radio-valves.

The Soviet state did engage in a certain amount of expectation-management. An eminent academician published a paper explaining that the happy citizens of the future would have all the shoes and socks and underwear they needed, "but this in no way presupposes superfluousness or extravagance". And First Secretary Khrushchev himself reproved intellectuals who might think the future held limitless "freedom" (which he clearly associated with sloppiness and disorder). "Communism is an orderly, organised society," he said in March 1963. "In that society, production will be organised on the basis of automation, cybernetics and assembly lines. If a single screw is not working properly, the entire mechanism will ground to a halt."

Yet the reason for insisting on the caveats was that the Soviet Union had gone ahead and promised Marx's plenty anyhow. Not as a vague aspiration for the future, either – not as a conveniently floating goal designed to keep the present hopeful. Nope: as a timed, detailed, schedule of events, with 1980 picked out as the date that the "material-technical" basis for full communism would be complete and the cornucopias would be switched on. The 1961 party congress adopted the imminent end of all scarcity as its official programme, thus making possibly the rashest and most falsifiable promise in the entire politics of the 20th century. An act so foolish can only be explained through idealism: Khrushchev's own, for he was a man whose troubled relationship with his conscience required a happy ending to give him retrospective absolution, but also the idealism coded despite everything into the structure of the régime. It was the same heedless true-belief at work which would manifest itself a generation later in Gorbachev.

The historian Stephen Kotkin describes the USSR as an edifice "booby-trapped with idealism", and that seems about right. The great grey tyranny ran on, in some sense depended on, hopes big enough to counterbalance the country's defects. Khrushchev really meant the promises that were spelled out with such excruciating frankness in the programme. Dialectical materialism was to imply denial and self-sacrifice no longer. The philosophy was going to pay off in the most literal and direct way; it was going to do what it said on the tin, and bring the materialists their material reward.

It was going to make first Russians and then all their friends the richest people in the world. Naturally this would involve zooming past the United States. "Today you are richer than us," Khrushchev had told a bemused dinner-party in the White House. "But tomorrow we will be as rich as you. The day after? Even richer!" Now, in 1961, he laid it all out, hour after hour, to an auditorium stuffed with delegates from all over Moscow's half of the cold-war globe. Soon, he told the assembled Cubans and Egyptians and East Germans and Mongolians and Vietnamese, Soviet citizens would enjoy products "considerably higher in quality than the best productions of capitalism". Pause a moment, and consider the promise being made there. Not products that were adequate or sufficient or OK; not products a little bit better than capitalism's. Better than the best. Considerably better. Ladas quieter than any Rolls-Royce. Zhigulis so creamily powerful they put Porsche to shame. Volgas whose doors clunked shut with a heavy perfection that made Mercedes engineers munch their moustaches in envy.

So the confidence that allowed Khrushchev to quip and hector and shoe-bang his way across the world stage was founded partly on a truth about the present, partly on a profound mistake about the future. That the Soviet dream didn't work out, that in 1980 Soviet citizens were not going to be strolling in the pleasure-garden of red plenty, we all know. (Khrushchev's own colleagues worked it out very quickly. They ousted him from the Politburo in the autumn of 1964 and consigned the 1961 programme to unmentionable oblivion.) What we've forgotten is that anyone ever took such a thing seriously; that it was ever anyone's sober expectation (or giddy expectation) that the grim, spartan one in the superpower duo was planning to win at hedonism.

Given that it was an error, a mirage, an astonishing mass delusion, what do we gain if we do remember it? Well, for a start, irony enough to glut even the greediest palate. Alongside our well-documented, well-founded knowledge that Soviet history was a tragedy ought to run a sense of it, too, as a comedy; a comedy of ideas and of things; a comedy in which material objects spin out of control, like the production line running awry in Chaplin's Modern Times, and refuse more and more catastrophically to play the roles assigned to them by bossy human intentions. Think of Laurel and Hardy pushing the piano up flight after flight of stairs until, right at the top, it gets away from them and slides right back down. That's the economic history of the Soviet Union in a nutshell: ascent, followed by pratfall.

But this shouldn't be the kind of comedy in which we laugh from a position of comfy security at the fools over there; and not just because the ascent of the Soviet piano was achieved at a monstrous price in human suffering. It should be the comedy of recognition we register, at this point in the early 21st century, when we're in mid-pratfall ourselves. Our own economic arrangements are currently generating not one but two complete sets of disastrous unintended consequences. Our failure to price the externalities of our energy use is baking the climate; our romantic indulgence of financiers has imploded our finances. We should be laughing at the Soviet disaster ruefully – with sympathy.

Don't get me wrong. The Soviet Union was a horrible society. Even once it had stopped purposely killing its citizens in large numbers, it oppressed them, it poisoned them with a toxic environment, it stuffed their ears continually with nonsense, it demanded their absolute passivity. It wasted their time. This last item sounds trivial. It wasn't. It had been one of the main points of the Marxist indictment of capitalism that it obliged people to bleed their labour-time into producing things they could feel no connection to, commodified things which had no real qualities except their price. Capitalism, Marx had argued, was a meaning-vampire, sucking away lives. Yet the Soviet attempt at an alternative came up with something worse: a form of work so divorced from usefulness that it condemned people to squander their finite store of weeks and months and years on churning out stuff you couldn't even be sure they were willing to pay for. By trying to concentrate directly on the use of things instead of their prices, the Soviet system lost hold of the one guarantee that anyone needed what was being manufactured. Result: futility, on the grand scale.

And when Soviet citizens went home from their pointless toil with their roubles in hand, they were then systematically disadvantaged as consumers. Soviet planners had done this deliberately at first, as a matter of strategy, to maximise the resources available for future investment, but under Khrushchev they tried to stop, and found they couldn't. The logic of the whole system compelled it. In a world where you'd get into trouble if you inconvenienced a factory waiting for its supply of widgets (so long as the factory had good enough connections), you could inconvenience a shopper looking for cheese with impunity, with no bad consequences at all. So the cheese, and the shopper, were always last on the list – an afterthought in an economy that was supposed to run entirely for human benefit. Contemporary joke: the phone rings at Yuri Gagarin's apartment and his little daughter answers it. "I'm sorry," she says, "Mummy and Daddy are out. Daddy's orbiting the earth, and he'll be back at 19:00 hours. But Mummy's gone shopping for groceries, so who knows when we'll see her again."

In turn the permanent state of shortage warped and deformed human relationships. The smooth impersonality of money-exchange in our society is so embedded that we take it absolutely for granted. If you've got the cash, you can have the thing. In the Soviet Union, having the cash was the mere beginning of the campaign to acquire the thing. Every transaction became personal, and not in a warm and fuzzy way. Since the scarce goods weren't rationed out by ability to pay, they were doled out in proportion to clout, influence, connections, ruthless calculations of mutual advantage. Soviet society was a tangled web of bullying, sycophancy, arm-twisting, back-scratching and emotional blackmail. Everyone made life as difficult as possible for those they dealt with, in order to be able to trade the easing of the difficulty for something else. You want a restaurant table, a dress, your phone repaired? Then find me some roofing felt, a Black Sea holiday, a private tutor for my son. Instead of post-capitalist freedom and sophistication, the Soviet Union offered pre-capitalist barter, with a large helping of robber baron-hood on the side.

The loudest and most important lesson of the Soviet experience should always be: don't ever do this again. Children, don't try this at home. Leave alone forever, please, this particular authoritarian recipe for bootstrapping a peasant society to wealth, because it only gets you halfway there, and leaves you surrounded by crumbling concrete and rusting machinery.

Yet we'd better remember to sympathise with the underlying vision that drove this disastrous history, because it is basically our own. As the ideological conflicts of the 20th century recede, it becomes clearer that the Soviet project for red plenty was just one in the 20th-century family of projects to hoick humanity out of its ancestral scarcity. The Soviet version is the cousin of ours; the loony cousin with blood "up to the elbows" (as Khrushchev put it, when asked in his forced retirement what he regretted most) but still one of the family. Through luck rather than virtue, for the most part, we happen to live in a variant that has succeeded better, so far. Our version isn't costless either. The steel and concrete required to sustain it are created for us elsewhere, out of sight, leaving us free to stroll around our pastel pavilion, on the side of which glimmers the word "Tesco". Inside are piled, just as Khrushchev hoped, riches to humble the kings of antiquity. But terms and conditions apply.

More surprisingly, there is something specific to sympathise with in the intellectual ambition of the Soviet moment. I'm sorry, you may say, thinking of the well-censored dullness of official Soviet thought – the what? Yes, for much of the 80 years during which the USSR was a unique experiment in running a non-market economy, the experiment was a stupid experiment, a brute-force experiment. But during the Soviet moment there was a serious attempt to apply the intellectual resources of the educated country the Bolsheviks had kicked and bludgeoned into being. All of the perversities in the Soviet economy that I've described above are the classic consequences of running a system without the flow of information provided by market exchange; and it was clear at the beginning of the 60s that for the system to move on up to the plenty promised so insanely for 1980, there would have to be informational fixes for each deficiency. Hence the emphasis on cybernetics, which had gone in a handful of years from being condemned as a "bourgeois pseudo-science" to being an official panacea.

The USSR's pioneering computer scientists were heavily involved, and so was the authentic genius Leonid Kantorovich, nearest Soviet counterpart to John Von Neumann and later to be the only ever Soviet winner of the Nobel prize for economics. Their thinking drew on the uncorrupted traditions of Soviet mathematics. While parts of it merely smuggled elements of rational pricing into the Soviet context, other parts were truly directed at outdoing market processes. The effort failed, of course, for reasons which are an irony-laminated comedy in themselves. The sumps of the command economy were dark and deep and not accessible to academics; Stalinist industrialisation had welded a set of incentives into place which clever software could not touch; the system was administered by rent-seeking gangsters; the mathematicians were relying (at two removes) on conventional neoclassical economics to characterise the market processes they were trying to simulate, and the neoclassicists may just be wrong about how capitalism works.

But if the horrible society of the Soviet Union left any legacy worth considering, if a pearl were ever secreted by the Soviet Union's very diseased oyster, this is it. And so follows the oddest implication of the Soviet moment. It may not be over. It may yet turn out to be unfinished business. For, from the point of view of "economic cybernetics", the market is only an algorithm. It is only one possible means of sharing out and co-ordinating economic activity: a means with very considerable advantages, in terms of all the autonomous activity and exploration of economic possibilities it allows, but not the only one, and not necessarily the best either, even at allowing autonomy and decentralisation. In the 20th century, devising the actual apparatus for a red plenty was an afterthought to the ideology. In the 21st century, it may be the algorithm that appears ahead of a politics to advocate it. In which case, the contest of plenties will be on again. And every year our processing power increases.

Francis Spufford's Red Plenty is published by Faber on 16 August.

• This article was amended on 11 August 2010. The original referred to the Cuban missile crisis in 1963. This has been corrected.