Lawn Road Flats in north London, also known as the Isokon building, is long and thin, with cantilevered exterior walkways that resemble the promenades on a ship. Built in 1934 by the Canadian architect Wells Coates, a follower of Le Corbusier, it was launched by the local MP, who smashed a bottle of beer (rather than champagne) on its rose petal pink facade. The four-story block, with its 32 “deck-access” apartments, was one of the first modernist buildings in Britain. According to one resident it had a “Brave New World air about it”, and indeed it aimed to be a pioneering showcase for a new way of living in a modern age. The “minimum” flats were fully furnished and serviced, and all occupants shared a laundry, communal kitchen and the Isobar. “As young men, we are concerned with a future that must be planned, rather than a past that must be patched up, at all costs,” wrote Coates. The idealistic developer Jack Pritchard, who commissioned the architect, shared this unshakable modernist belief “that a rational, brave new approach to all problems would make for a better world”. Pritchard hoped similar buildings would soon replace working-class tenements all over London.

The Isokon was a modernist utopia made concrete, and its bohemian, leftist residents reflected many of the political battles of the age. The building was inspired by Pritchard and Coates’s visit to see Mies van der Rohe’s Weissenhof Estate in Stuttgart, a model community that included houses by Mies, alongside others by modernist pioneers such as Walter Gropius and Le Corbusier. The previous year Mies had built a controversial brick monument to the martyred German communist leaders Rosa Luxemburg and Karl Liebknecht, and his answer to the problem of mass housing was a socialist manifesto of sorts. Pritchard and Coates also made a pilgrimage to the Bauhaus school in Dessau, which had been directed by Gropius and then Mies, but was now deserted after having being forcibly closed by the Nazis, its staff accused of fostering “cultural bolshevism”. Pritchard offered refugees from the Bauhaus free accommodation in the Isokon. These included Gropius, the furniture maker Marcel Breuer, and the artist László Moholy-Nagy, who soon began work on the special effects for Alexander Korda’s film of HG Well’s utopian novel The Shape of Things to Come. Gropius would gallantly defend the “handsome” Isokon (now Grade I-listed) when the literary magazine Horizon named it “the ugliest building in London”.

The Isokon building in Hampstead, London. Photograph: Daniel Lynch/REX/Shutterstock

As David Burke reveals in his book The Lawn Road Flats, in the 1930s the Isokon was also home to a succession of Soviet spies, the most prominent of which was Arnold Deutsch, the cousin of Oscar Deutsch who owned the Odeon cinema chain. Once an employee of the psychoanalyst and sexual radical Wilhelm Reich, who dreamed of a “genital utopia”, Deutsch had been sent to England by the Comintern and instructed “to cultivate young radical high-fliers from leading British universities before they entered the corridors of power”. Deutsch lived in Flat 7, from which he recruited 20 agents, including Kim Philby, Anthony Blunt and the rest of the “Cambridge five”. Indeed, no fewer than 11 Soviet spies lived at Lawn Road between the mid-1930s and mid-40s. (By coincidence, Agatha Christie also lived for six years in the Isokon, where she wrote her only spy novel, N or M?) The building, with its international community and communitarian ethos, was not only a fitting statement of their utopian ideological beliefs, but it turned its back to the street, giving them and their informants discreet access to Belsize Park tube through the adjoining wood.

When we launched the London Design Biennale, of which I’m director, we chose as the inaugural theme, Utopia by Design, to celebrate the 500th anniversary of the publication of Thomas More’s classic, Utopia, and to reflect on the rich, contentious history of the modernist design it inspired. The Bauhaus, for example, in homage to More, named their publishing house the Utopia Press. In September this year, design teams from more than 30 countries will exhibit ambitious installations at Somerset House that explore how architecture, design and engineering might contribute in some small way to making the world a better place and our cities more livable. Utopia is a particularly urban invention: as Lewis Mumford observed, “the first utopia was the city itself”, which represented man’s technological triumph over nature. More’s Utopia – a neologism meaning both “good place” and “no place” – is a man-made island with 54 identical cities that are described as a triumph of design. Led by the enlightened Utopos, the Platonic philosopher-king after which the island took its name, the citizens of Utopia reject the idea of money or private property, work a six-hour day, share everything, are without greed and pride, and place a premium on human happiness.

More intended his imaginary ethnography to critique the status quo by presenting a radical alternative. It was, in More’s words, “a fiction whereby the truth, as if smeared with honey, might a little more pleasantly slide into men’s minds”. His satirical novel, which has never since been out of print, has spawned a vast literary genre, and has exerted an unprecedented influence on architecture, design and the form of our built environment. From the ideal cities of the Renaissance, to Ebenezer Howard’s garden city movement, to Le Corbusier’s modernist City of Tomorrow, with its “single society, united in belief and action”, design was seen as a critical tool with which it was possible to transform both political reality and the quality of life. Architecture, the most utopian of the arts, was interpreted as a harmonising force, able to shape not only space, but to use technology to mould attitudes and beliefs for the better.

Nowhere was this blind faith in technology celebrated more than in World Fairs, which ever since the 1851 Crystal Palace exhibition, have been seen as a catalyst for international cooperation and peace. Countries and large corporations spent enormous sums promoting their values and aspirations through architecture, design and technology at these international pageants. Against the background of the depression, the 1939 World of Tomorrow fair in New York reflected this technological optimism. Exhibits for Chrysler and General Motors by the industrial designers Raymond Loewy and Norman Bel Geddes presented panoramic visions of the city of the future and its avant-garde transport systems. The GM pavilion was a roller coaster ride over a kinetic vision of the atomic US of the 1960s – a monumental landscape punctuated by glass domes, revolving airports and, in tune with its sponsor, seven-lane superhighways. On exiting, visitors were given a badge that read: “I Have Seen the Future”. In 1964, with Futurama II, GM updated the exhibition to show underwater cities, as well as urban conglomerations in a deforested Amazon, in Antarctica and on the moon. Utopia was the new frontier, even if it was still just around the corner.

Candidates for the Biosphere II project line up on the ‘lung’ of its enclosed environment. Photograph: Roger Ressmeyer/Corbis

In his recent book Last Futures: Nature, Technology and the End of Architecture, Douglas Murphy presents a fascinating history of these failed utopian architectural schemes, in which he nevertheless sees “glimpses of hope”, and from which he believes lessons can be learned as we face current crises. Many of these fantastic, futuristic proposals placed the world under glass, like Buckminster Fuller’s outsized geodesic dome did at Expo 67 in Montreal, almost as if to preserve our fragile ecosystem in amber. The most extravagant of these space-frame structures was Biosphere 2, built in the early 1990s at a cost of $150m, which Douglas describes as “the last countercultural blast in architecture”. Situated in the Arizona desert, it was a hermetically sealed, three-acre complex of Mesoamerican pyramids and geodesic domes that was designed as a self-regulating, man-made paradise. Biosphere 2 (the first being Earth) was a high-tech experiment inspired by ecological fear to see if the colonisation of space might one day be possible. Eight “bionauts”, as they called themselves, lived for two years locked inside this crystalline, nuclear bunker. Its huge vaulted structures contained a tropical rain forest, a grassland savannah, a mangrove wetland and salt-water ocean, complete with coral reef – everything one might possibly need to weather the apocalypse. It was described in the press as a “planet in a bottle”, “Eden revisited” and “Greenhouse Ark”.

Despite being schooled in The Whole Earth Catalog and studying manuals with titles such as “How to Grow More Vegetables Than You Ever Thought Possible on Less Land Than You Can Imagine”, yields harvested in the Biosphere were disappointing and the bionauts began to starve. They lost 18% of their body weight, mostly in the first six months. Food was carefully measured into equal portions, and the larder had to be locked because it was being raided by the ravenous bionauts. An infestation of ants and cockroaches managed to get through, nevertheless, further depleting precious stocks. When carbon dioxide levels rose, extra oxygen had to be pumped in to keep the inhabitants alive. Tourists came in thousands to peer into the vivarium at the laboratory rats – the Biosphere was an acknowledged precursor to the Big Brother reality-TV show. The emaciated pioneers, engaged in a power struggle, split into two factions who scarcely spoke to each other. By the time they emerged, the utopia they envisaged had descended into chaos, and the experiment was acknowledged as a failure. Time magazine would call it one of the “50 worst ideas of the 20th century”.

As modernist architects and designers pursued social perfection with uncritical zeal, utopian ideals often degenerated into dystopian realities. Writers such as Orwell, HG Wells and Aldous Huxley illustrated the dangers inherent in utopian thinking, and questioned the utopian faith in science and technology as an industrial lifeboat that promised to banish scarcity and waste. In novels such as Brave New World and Nineteen Eighty-Four, the regimented technocratic order, where one political model satisfies all, is shown to be totalitarian and repressive. Correspondingly, the ideological pursuit of utopias fell out of fashion. In the famous “kitchen debate” at the 1959 American National Exhibition in Moscow, which followed a Soviet exhibition in New York earlier that year, Khrushchev and Nixon used industrial design to argue the respective benefits of their communist and capitalist utopias. However, by 1989 with the fall of the Berlin Wall, Margaret Thatcher could celebrate the end of ideology with the claim that “there is no alternative”. Biosphere 2 was just an expensive exercise in space race nostalgia.

Extra oxygen had to be pumped in to the Biosphere to keep the bionauts alive, and roaches infested their food supplies

Utopianism became a term of abuse, shorthand for “hopelessly idealistic”. The rightwing philosopher Roger Scruton dismissed the “utopian fallacy”, which was “rooted not in intellect, but in emotional needs, which leads to the acceptance of absurdities”. Unlike such idealistic, escapist fantasies, the apocalyptic visions of anti-utopias or dystopias seemed to engage more with the problems of the real world. The Isokon dream had run aground; the modernist housing schemes that the Bauhaus and Coates hoped might transform society had become hotbeds of vandalism and crime, and the prevalent mood was of pessimistic nihilism. In Archaeologies of the Future (2005), Fredric Jameson would call for an “anti-anti-utopianism”, lamenting that it was now “easier to imagine the end of the world than an alternative to capitalism”. Can something be salvaged from utopian thinking, a drive that the philosopher Ernst Bloch referred to as “the principle of hope”? The utopian impulse allows us to escape the blinkers of the present and dream, telling stories about alternative futures that ask important questions about the world in which we live. The London Design Biennale will feature some of these visions, which aim to provoke real change by suggesting inspiring or cautionary futures.

“A map of the world that does not include utopia is not even worth glancing at,” quipped Oscar Wilde, “for it leaves out the country at which Humanity is always landing. And when Humanity lands there, it looks out, and, seeing a better country, sets sail. Progress is the realisation of Utopias.” In the frontispiece of More’s Utopia, alongside the alphabet he invented for the nation, is a map depicting his fictional island. It makes knowing reference to the medieval maps of the world, which often showed the physical site of paradise. Eden was sometimes near the Euphrates, or in sub-Saharan Africa, or on a remote island, or drowned by the Pacific; it was often presented as a walled fortress, situated in real geography but forever closed off to us. The engraving depicting the topography of Utopia is by Ambrosius Holbein, the elder brother of Hans Holbein, who painted a portrait of More and whose The Ambassadors, with its famous anamorphic skull, hangs in the National Gallery. With its “ship of teeth” in the foreground, the map of Utopia also resembles a camouflaged human skull. More’s Utopia functions as a vanitas or memento mori, reminding us of the fleetingness of worldly pleasures, and that our quest for an earthly paradise is always doomed. But, he hints, perhaps we should still die trying.

• The London Design Biennale will take place from 7 to 27 September at Somerset House, the Strand, London WC2. londondesignbiennale.com