A quarter of a century ago, the whole idea of utopia seemed irredeemably sullied. At the start of the 1990s, the largest social experiment in human history – the USSR – imploded, and with it went the notion that imagining a radically different society was a serious activity. It seemed that the rewards of such experiments were always so enticing that genocide inevitably ensued.

That was the lesson drawn from any totalitarian regime informed by the highest (or lowest) idealism: the Khmer Rouge, the Videla regime in Argentina, Nazi Germany, you name it. Back then, it was thought best not to fantasise too much about a better world, but to learn to live in this one. The academic and political atmosphere in the 1990s was decidedly pragmatic, rather than optimistic. It was an era in which the liberal democracies celebrated (prematurely, of course) “the end of history”. The story of humanity was a march to freedom, we were told, and we had arrived. This was as good as it got, and the idealists and unrealists should stop fantasising, because it was a dangerous hobby.

The Farm … a still-active Tennessee community founded in 1971 by Stephen Gaskin and 400 hippies

Everything looks different now. George Bush Sr’s new world order is frightening and deeply disordered. Religion, which sociologists predicted would slowly slide out of view, is the dominant political issue of the early 21st century, a form of utopianism that just won’t go away. Meanwhile capitalism, which was the motif of triumphalist freedom, seems less noble after Enron, Madoff, Libor and RBS. If anything, people are even more fed-up with the laziness, injustices and profligacy of consumerism than they were back in the 1990s.

In 1999, John Carey published a great compendium of excerpts on the theme of utopias and dystopias. They grow, he wrote, “from desire and fear … cry out for our sympathy and attention, however impractical or unlikely they appear”. To live without that alternative to contemporary society was to remove from record the dreams and nightmares of the human subconscious. To live without such a record was to live in a bland, thin world.

Different worlds … Gulliver’s Travels Photograph: Allstar/Valeness-Belvision

Carey made clear in his opening pages that the alternative to bland was often bloody; that because writers had the fig-leaf of fiction, they were able to toy with ideas which might otherwise seem immoral or icily rational – genocide, of course, but also eugenics, free love, the abandonment of family, the eradication of money, of monogamy, of private property and so on.

Genocide has always been there in these otherworldly narratives: in Gulliver’s Travels, the Houyhnhnms wanted to wipe out the Yahoos; Gerrard Winstanley, founder of the True Levellers, recommended the execution of all lawyers. There are few utopias that don’t blithely eradicate unwanted elements, and while sometimes the author’s sympathies are clear (Winstanley was pretty black-and-white) others have satirical immunity, presenting their fiction not as a window on to a better future, but as a mirror held up to bleak humanity. It was often hard to tell whether the fiction was an aspiration or a warning.

Utopia by Thomas More

That, perhaps, is why the genre remains so appealing: it has always been slightly smoke-and-mirrors, a bit of a guessing game about how serious the proposition actually was. Many understandably believe that “utopian” is shorthand for dreamy impracticality. Take the notion that humans might one day be so good that lies are despised (imagined in Edward Bellamy’s 1888 science-fiction novel Looking Backward) or taxes willingly overpaid, as in Louis-Sebastien Mercier’s Memoirs of the Year 2500: “I saw several people, with easy, cheerful, contented looks, throw sealed packets into the chest, as in our day they threw letters into the post office.” Some of Mercier’s imaginings, which were published in 1770, appear all too removed from human avarice and from what policy wonks call “deliverability”.

But utopianism is an academic parlour game precisely because it can be divisive, entertaining and challenging. And with the quincentenary of Thomas More’s Utopia falling this year, the parlour game is again being dusted off. On 25 January, London’s Somerset House begins 12 months of exhibitions, installations and commissions to investigate the renewed allure of utopianism. One of the most eye-catching events is the opportunity to see the view from Anarres, the anarchic planet in one of the last century’s greatest novels, Ursula K Le Guin’s The Dispossessed. In that interplanetary book, the arch-anarchist Shevek travels to a new planet, and struggles to understand the “propertarian” anxieties: “Was it because, no matter how much money they had, they always had to worry about making more, lest they died poor?”

It feels as if the year’s events are planned to make people ponder seriously what it means, in our confused age, to be countercultural. What would utopian dreaming, in 2016, look like? Would our concerns be the same as those of More, or very different? Presumably the impossible dream of many is now the redemption of a scarred and suffocating planet; or maybe global peace or the eradication of religion. That interface between politics and utopianism is where the game gets interesting. Utopianism is like the manifesto we would write if no one were watching, if all the rules could be rewritten by just one, benevolent dicatator.

Banned … HG Wells’s In the Days of the Comet proposed free love. Photograph: Popperfoto/Getty Images

And it’s surprising how, over the centuries, all those literary dictators have raised very comparable issues. One constant has been the vexed question of the family. From Plato onwards, writers have fantasised about a universal brotherhood in which blood allegiances are replaced by infinite goodwill. Sometimes the renunciation of family isn’t selfless, but simply about freedom and, specifically, free love. HG Wells’s In the Days of the Comet suggested that sex with whoever you wanted was the future of a reasoning society (and it duly got banned from public libraries).

Since utopianism is the fantasist’s literary genre of choice, it’s not surprising that so many utopias were given over to fantasies about orgiastic rituals, like this from Charles Fourier’s Harmony: “When the Head Fairy waves her wand a semi–bacchanalia gets under way. The members of both groups rush into each other’s arms, and in the ensuing scramble caresses are liberally given and received. Everyone strokes and investigates whatever comes to hand and surrenders himself or herself to the unfettered impulses of simple nature. Each participant flits from one person to another, bestowing kisses everywhere with as much eagerness as rapidity.”

But it’s in the eradication of money and private property that utopianism most clearly intrigues the orphans of radical politics. Those who still devoutly believe in common ownership can’t get enough of utopianism that promises equality, from Henry George’s Progress and Poverty (extolling the notion of a single land tax) to Gracchus Babeuf’s egalitarianism in all things (“education is a monstrosity when it is unequally shared”). Both were working in that fertile ground shared by utopians and politicians, where radical dreams might just germinate in the soil of the real world.

Embroiderers in the 1930s at the workshop begun by William Morris half a century before in Merton Abbey Mills, London. Photograph: Fox Photos/Getty Images

Many writers, of course, have gone on to be – in major and minor ways – utopian pioneers. Nathaniel Hawthorne was one of the founders of a Fourier-inspired Brook Farm (and wrote The Blithedale Romance as a result); Robert Owen had published seven books by the time he set up New Harmony in Indiana. William Morris was forever dreaming up, and putting into practice, better ways of living, as well as writing News From Nowhere, another utopian fantasy.

But when the game moves from something on paper to a reality put on land with non-fiction human beings, there are often disasters. The Brixton cult where the Maoist cult leader Aravindan Balakrishnan raped and abused followers is just the most recent and glaringly horrible example of what happens when a dictator gets going. News stories have forever reiterated the pitfalls of utopianism. Whether it was Jim Jones in Guyana or David Koresh in Waco, it has seemed pretty plain that intentional communities of idealists often go fatally berserk. Books like My Life In Orange, or Frances Fitzgerald’s magisterial dismantling of Rajneeshpuram in Cities on a Hill, have underlined the point that experimentation with the norms of society is like putting machine-guns in a nursery.

Waco, Texas, where 76 people died following the siege of a religious cult led by David Koresh. Photograph: Bob Daemmrich/AFP/Getty Images

But often the results are merely chaotic or eccentric. The 20,000 acres of Owen’s Harmony lacked the leadership and direction of its Lutheran predecessors and slowly petered out. The Shakers, in contrast, didn’t lack leadership but progeny: their response to the gospel was to ban all sexual intercourse. The Fruitlands Community, where Louisa May Alcott of Little Women fame grew up, was troubled not just by sex but also, honestly, by vegetables that grew downwards.

But for all our sniggering about communal weirdness, many of the great advances in human compassion have come about because of utopian experimentation. Whether or not you like what they stand for, monasteries have lived out the core utopian ideals for centuries: swap celibacy for free love, and you’ve got all the classic ingredients – no family, no private property, no punishment (because of forgiveness).

Or take The Farm, the Tennessee community founded in 1971 by my hero, Stephen Gaskin. Early pioneers took vows of poverty, enjoyed sex but only within monogamous relationships, and thought cannabis a sacrament (while prohibiting hard drugs). It might sound old-school hippie, but Gaskin was a devout man who created a charitable arm called Plenty, which became a relief organisation working in Guatemala, New Orleans and so on.

Fruitlands, Massachusetts, where Little Women author Louisa May Alcott lived in the 1840s.

One naturally assumes that mega-riches are needed to start one of these communities, but actually the reverse is often true. There was a beautiful book published years ago called Sweet Earth. It was a series of photographs by Joel Sternfeld subtitled Utopian Experiments in America. What emerged from the stunning images of ploughs, compost loos and earth-ships was the fact that such spaces often grow not on rich soil, but on barren, impoverished places like Slab City.

It’s an old truism that if you want generosity, go to the person who has nothing. The same message emerges clearly from the pages of Rebecca Solnit’s A Paradise Built in Hell: the greatest, anarchic solidarity emerges in the most desperate and damaged places – earthquakes, warzones and so on. As Le Guin wrote in The Dispossessed: “I’m trying to say what I think brotherhood really is. It begins – it begins in shared pain.” Egalitarianism, that old utopian aspiration, can only happen amongst long-suffering commoners.

In the end, one’s attitude to utopianism usually depends on one’s understanding of progress, since progress is, as Wilde wrote, just “the realisation of utopias”. Those who think human history is nihilistic chaos will be pretty snooty about utopians. But those whigs, liberals, egalitarians, eco-warriors, freedom-fighters or pacifists who are able to imagine a far better world all hear the call of utopianism. The key question, of course, is whether one’s response to that call leads to killing or to compassion.