In 1959, if you were walking the sand dunes near Florence, Oregon, you might have encountered a burly, bearded extrovert, striding about in Ray-Ban Aviators and practical army surplus clothing. Frank Herbert, a freelance writer with a feeling for ecology, was researching a magazine story about a US Department of Agriculture programme to stabilise the shifting sands by introducing European beach grass. Pushed by strong winds off the Pacific, the dunes moved eastwards, burying everything in their path. Herbert hired a Cessna light aircraft to survey the scene from the air. “These waves [of sand] can be every bit as devastating as a tidal wave … they’ve even caused deaths,” he wrote in a pitch to his agent. Above all he was intrigued by the idea that it might be possible to engineer an ecosystem, to green a hostile desert landscape.

About to turn 40, Herbert had been a working writer since the age of 19, and his fortunes had always been patchy. After a hard childhood in a small coastal community near Tacoma, Washington, where his pleasures had been fishing and messing about in boats, he’d worked for various regional newspapers in the Pacific northwest and sold short stories to magazines. He’d had a relatively easy war, serving eight months as a naval photographer before receiving a medical discharge. More recently he’d spent a weird interlude in Washington as a speechwriter for a Republican senator. There (his only significant time living on the east coast) he attended the daily Army-McCarthy hearings, watching his distant relative senator Joseph McCarthy root out communism. Herbert was a quintessential product of the libertarian culture of the Pacific coast, self-reliant and distrustful of centralised authority, yet with a mile-wide streak of utopian futurism and a concomitant willingness to experiment. He was also chronically broke. During the period he wrote Dune, his wife Beverly Ann was the main bread-winner, her own writing career sidelined by a job producing advertising copy for department stores.

Soon, Herbert’s research into dunes became research into deserts and desert cultures. It overpowered his article about the heroism of the men of the USDA (proposed title “They Stopped the Moving Sands”) and became two short SF novels, serialised in Analog Science Fact & Fiction, one of the more prestigious genre magazines. Unsatisfied, Herbert industriously reworked his two stories into a single, giant epic. The prevailing publishing wisdom of the time had it that SF readers liked their stories short. Dune (400 pages in its first hardcover edition, almost 900 in the paperback on my desk) was rejected by more than 20 houses before being accepted by Chilton, a Philadelphia operation known for trade and hobby magazines such as Motor Age, Jewelers’ Circular and the no-doubt-diverting Dry Goods Economist.

Kyle MacLachlan in director David Lynch’s film adaptation of Dune. Photograph: Allstar/UNIVERSAL

Though Dune won the Nebula and Hugo awards, the two most prestigious science fiction prizes, it was not an overnight commercial success. Its fanbase built through the 60s and 70s, circulating in squats, communes, labs and studios, anywhere where the idea of global transformation seemed attractive. Fifty years later it is considered by many to be the greatest novel in the SF canon, and has sold in millions around the world.

***

Dune is set in a far future, where warring noble houses are kept in line by a ruthless galactic emperor. As part of a Byzantine political intrigue, the noble duke Leto, head of the Homerically named House Atreides, is forced to move his household from their paradisiacal home planet of Caladan to the desert planet Arrakis, colloquially known as Dune. The climate on Dune is frighteningly hostile. Water is so scarce that whenever its inhabitants go outside, they must wear stillsuits, close-fitting garments that capture body moisture and recycle it for drinking.

The great enemy of House Atreides is House Harkonnen, a bunch of sybaritic no-goods who torture people for fun, and whose head, Baron Vladimir, is so obese that he has to use little anti-gravity “suspensors” as he moves around. The Harkonnens used to control Dune, which despite its awful climate and grubby desert nomad people, has incalculable strategic significance: its great southern desert is the only place in the galaxy where a fantastically valuable commodity called “melange” or “spice” is mined. Spice is a drug whose many useful properties include the induction of a kind of enhanced space-time perception in pilots of interstellar spacecraft. Without it, the entire communication and transport system of the Imperium will collapse. It is highly addictive, and has the side effect of turning the eye of the user a deep blue. Spice mining is dangerous, not just because of sandstorms and nomad attacks, but because the noise attracts giant sandworms, behemoths many hundreds of metres in length that travel through the dunes like whales through the ocean.

Have the Harkonnens really given up Dune, this source of fabulous riches? Of course not. Treachery and tragedy duly ensue, and young Paul survives a general bloodbath to go on the run in the hostile open desert, accompanied, unusually for an adventure story, by his mum. Paul is already showing signs of a kind of cosmic precociousness, and people suspect that he may even be the messiah figure foretold in ancient prophecies. His mother, Jessica, is an initiate of the great female powerbase in an otherwise patriarchal galactic order, a religious sisterhood called the Bene Gesserit. Witchy and psychically powerful, the sisters have engaged in millennia of eugenic programming, of which Paul may be the culmination.

Frank Herbert. Photograph: Ulf Andersen/Getty Images

This setup owes something to the Mars stories of Edgar Rice Burroughs and Isaac Asimov’s Foundation books, as well as the tales written by Idaho-born food chemist Elmer Edward “Doc” Smith, creator of the popular Lensman space operas of the 1940s and 50s, in which eugenically bred heroes are initiated into a “galactic patrol” of psychically enhanced supercops. For Smith, altered states of consciousness were mainly tools for the whiteous and righteous to vaporise whole solar systems of subversives, aliens and others with undesirable traits. Herbert, by contrast, was no friend of big government. He had also taken peyote and read Jung. In 1960, a sailing buddy introduced him to the Zen thinker Alan Watts, who was living on a houseboat in Sausalito. Long conversations with Watts, the main conduit by which Zen was permeating the west-coast counterculture, helped turn Herbert’s pacy adventure story into an exploration of temporality, the limits of personal identity and the mind’s relationship to the body.

Every fantasy reflects the place and time that produced it. If The Lord of the Rings is about the rise of fascism and the trauma of the second world war, and Game of Thrones, with its cynical realpolitik and cast of precarious, entrepreneurial characters is a fairytale of neoliberalism, then Dune is the paradigmatic fantasy of the Age of Aquarius. Its concerns – environmental stress, human potential, altered states of consciousness and the developing countries’ revolution against imperialism – are blended together into an era-defining vision of personal and cosmic transformation.

Books read differently as the world reforms itself around them, and the Dune of 2015 has geopolitical echoes that it didn’t in 1965, before the oil crisis and 9/11. Remember that European beach grass binding together those shifting dunes? Paul Atreides is a young white man who fulfils a persistent colonial fantasy, that of becoming a God-king to a tribal people. Herbert’s portrayal of the “Fremen” (the clue’s in the name) owes much to TE Lawrence and Wilfred Thesiger’s enthusiastic portrayals of the Bedouin of Arabia’s Empty Quarter. Fremen culture is described in words liberally cribbed from Arabic. They go on “razzia” raids, wear “aba” and “bourka” robes, fear a devil called “Shaitan” and so on. They are tough, proud and relatively egalitarian. The harshness of their environment has given them an ethic of fellowship and mutual aid. They are what Kipling would have termed “one of the martial races”: absolutely to be admired, possessing none of the negative “oriental” traits – deviousness, laziness and the like. They are, however, not carbon-copy Bedouin: Herbert freely mixes elements of Zen into their belief system, and also, intriguingly, suggests that their messianic eschatology – the sense in which they were “waiting” for Paul – may have been seeded in previous millennia by the Bene Gesserit order as part of its murky eugenic plans. Herbert, whose female characters are consistently strong and active, has also ditched the strict sexual divisions of actually existing Bedouin culture. Thus Fremen women do their share of fighting and fearlessly contradict their menfolk, though there is still a fair amount of child-bearing and housework to be done while the men are off riding worms.

Dune official trailer (1984)

What makes Dune more palatable than, say, the gruesome spectacle of a blonde-wigged Emilia Clarke carried aloft by ethnically indeterminate brown slaves in Game of Thrones, is the sincerity of Herbert’s identification with the Fremen. They are the moral centre of the book, not an ignorant mass to be civilised. Paul does not transform them in his image, but participates in their culture and is himself transformed into the prophet Muad’Dib. If Paul is one-part Lawrence of Arabia, leading his men on to Aqaba, he is also the Mahdi. Dune glosses this word as “in the Fremen messianic legend, The One Who Will Lead Us into Paradise”. In Islamic eschatology, the honorific Mahdi has a long and complex history. Various leaders have claimed or been given it. Most Shia identify the Mahdi with the 12th or Hidden Imam, who will imminently reveal himself and redeem the world. To the British, it will always be the name of the warrior prophet who swept through the Sudan in the 1880s, killing General Gordon on the steps of the palace in Khartoum and inspiring a thousand patriotic newspaper etchings. As Paul’s destiny becomes clear to him, he begins to have visions “of fanatic legions following the green and black banner of the Atreides, pillaging and burning across the universe in the name of their prophet Muad’Dib”. If Paul accepts this future, he will be responsible for “the jihad’s bloody swords”, unleashing a nomad war machine that will up-end the corrupt and oppressive rule of the emperor Shaddam IV (good) but will kill untold billions (not so good) in the process. In 2015, the story of a white prophet leading a blue-eyed brown-skinned horde of jihadis against a ruler called Shaddam produces a weird funhouse mirror effect, as if someone has jumbled up recent history and stuck the pieces back together in a different order.

***

After Dune was published, Herbert, the consummate freelancer, kept a lot of irons in the fire. He wrote about education for the Seattle Post-Intelligencer and lectured at the University of Washington. In 1972, during the American push to extricate itself from the south-east Asian quagmire, he worked in Vietnam, part of a project called “Land to the Tiller”, aimed at cutting Viet Cong recruitment by enacting land reform. He built a family home on the Olympic peninsula which he thought of as an “ecological demonstration project”. He built his own solar collector, wind plant and methane fuel generator. In a 1981 interview he described himself a “technopeasant”. As the cult of Dune took off during the 1970s, he wrote a series of increasingly convoluted sequels, following Paul’s descendants as they fulfilled the cosmic destiny of the Atreides line. Since his death in 1986, his son and another writer have produced a further 13 books.

Star Wars: Episode IV - A New Hope (1977). Photograph: Allstar/Lucasfilm

By rights, Dune ought to have become a big movie. An attempt by the visionary Chilean film maker Alejandro Jodorowsky to bring it to the screen became one of the great “what if” stories of SF cinema. Jodorowsky had extraordinary collaborators: visuals by Moebius and HR Giger, spaceships designed by the English illustrator Chris Foss. Orson Welles was to play Baron Harkonnen, Salvador Dali the Emperor. Pink Floyd and Magma were on board to do the soundtrack. But Jodorowsky’s prog-tastic project was strangled in the crib by risk-averse Hollywood producers. After a period of film industry bloodletting, David Lynch shot a version in 1984, only for Universal to release a cut that he hated so much he had his name removed from the credits. Lynch’s film is actually much better than its terrible reputation, but Sting in a codpiece and a Toto soundtrack will never match the potential greatness of Jodorowsky’s unmade epic.

Actually, the great Dune film did get made. Its name is Star Wars. In early drafts, this story of a desert planet, an evil emperor, and a boy with a galactic destiny also included warring noble houses and a princess guarding a shipment of something called “aura spice”. All manner of borrowings from Dune litter the Star Wars universe, from the Bene Gesserit-like mental powers of the Jedi to the mining and “moisture farming” on Tattooine. Herbert knew he’d been ripped off, and thought he saw the ideas of other SF writers in Lucas’s money-spinning franchise. He and a number of colleagues formed a joke organisation called the We’re Too Big to Sue George Lucas Society.

Though in his later years he enjoyed huge success, Herbert, the man who dreamed of greening the desert, had mixed feelings about the future. In Dune, he has Kynes, the “First Planetologist of Arrakis” (and hero of the novel’s first draft) muse that “beyond a critical point within a finite space, freedom diminishes as numbers increase. This is as true of humans in the finite space of a planetary ecosystem as it is of gas molecules in a sealed flask. The human question is not how many can possibly survive within the system, but what kind of existence is possible for those who do survive.” Gloomy Malthusianism was much in vogue in the 1960s and 70s. In 1968 Paul Ehrlich’s The Population Bomb became a runaway bestseller, predicting mass starvation unless population growth was restricted. The flip side of the green movement’s valorisation of small scale and self-reliance is an uneasy relationship with the masses, and with the idea of economic growth more generally. Herbert’s libertarian politics reinforced this worry. In Dune, Paul knows that if the desert planet is made to bloom, it will support a larger population, and the ethic of individualism will be eroded. He himself, as he is transformed from aristocrat to messiah, loses his individuality and begins to dissolve into myth, becoming part of a Jungian collective unconscious. But perhaps Herbert would take heart from the thought that history does not appear to be teleological and some long-term plans do not take on the character of destiny. Fifty years after Dune’s publication, the US Department of Agriculture is still at work on the Oregon Dunes, rooting out European beach grass, an “invasive non‑native species”. They want to return the dune processes to their natural state.