JG Ballard’s High-Rise, published 40 years ago and soon to be seen on cinema screens in a film adaptation directed by Ben Wheatley, begins with one of the most arresting first sentences in 20th-century literature: “Later, as he sat on his balcony eating the dog, Dr Robert Laing reflected on the unusual events that had taken place within this huge apartment building during the previous three months.”

High-Rise is the final part of a quartet of novels – the first three are The Atrocity Exhibition (1970), Crash (1973) and Concrete Island (1974) – with each book seeded in the previous one. Thematically High-Rise follows on from Concrete Island with its typically Ballardian hypothesis: “Can we overcome fear, hunger, isolation, and find the courage and cunning to defeat anything that the elements can throw at us?” What links all of them is the exploration of gated communities, physical and psychological, a theme that is suggestive of Ballard’s childhood experiences interned by the Japanese in a prisoner-of-war camp on the outskirts of Shanghai in the 1940s. It was, he always claimed, an experience he enjoyed.

The built environment is not a backdrop, rather it is integral and distinctive in its recurring imagery – from abandoned runways, to curvilinear flyovers and those endlessly mysterious drained swimming pools. Perhaps more than any other writer, he focused on his characters’ physical surroundings and the effects they had on their psyches. Ballard, who died in 2009, was also interested in the latent content of buildings, what they represented psychologically. Or, as he once obliquely put it, “does the angle between two walls have a happy ending?” – by which he meant that we project narrative on to external reality, that the imagination remakes the world. In Ballard’s fiction, nothing is taken at face value.

In High-Rise and Concrete Island especially, Ballard examines the flip side to what he called the “overlit realm ruled by advertising and pseudo-events, science and pornography” that The Atrocity Exhibition and Crash mapped out. Under-imagined or liminal spaces, such as multi-storey car parks and motorway flyovers, act as metaphors for the parts of ourselves that we ignore or are unaware of. His characters are often forced to assess the physical surroundings and, by extension, themselves rather than to take them for granted.

Chris Hall on Ballardian architecture

Ballardian space – what he called “inner space” to differentiate it from the science fiction that concerned itself with distant planets and space rockets – is in fact a fusion of inner and outer space. There is no “out there” totally separate from his characters; just as there is no exclusively private, isolated inner life. His most psychologically fulfilled characters look to transcend their physical surroundings, however hostile, by embracing them.

The obsessive manner in which Ballard came to use the built environment in this way began with his short story “The Terminal Beach” in 1964, in which a man called Traven finds himself on an abandoned atomic testing site on a Pacific atoll after his wife and young son have died in a car crash. The abstract lexicon in the story evokes a prison – there are mazes, blocks, bunkers, cells, corridors, aisles. His mind jumps from one fractured event to another in a kind of short circuit. Time becomes “quantal” just like the blocks on the island. There is no past, no future – just an endless, eventless present. He chooses to stay there with the ever intensifying hallucinations of his dead family rather than be rescued, and he hides from a search team when they come on to the island. Traven doesn’t so much embrace his surroundings as become them.

In Concrete Island, Ballard maroons the architect Robert Maitland, Crusoe-like, in a triangular interzone of a motorway intersection, armed only with “a tool-kit, some architectural journals and six bottles of white Burgundy”. The situation he finds himself trying to escape is an extended metaphor for Maitland’s personal life, trapped in the dead space between himself, his wife and his mistress.

Ballard kept repeating his mantra that “In a totally sane society, madness is the only freedom”. In the chilling novella Running Wild (1988), set in a suburban gated community in Berkshire, the lives of the residents are a paralysing middle-class carousel of ordered sterility. Ballard details how this terminal boredom leads affectless children to kill their parents – and get away with it.

In High-Rise, over the course of three months, a 40-storey tower block housing 2,000 residents – “a small vertical city” – descends from civilisation to tribalism to hunter-gatherer savagery (there is even a suggestion of cannibalism), in a kind of mass psychosis where they retreat from the outside world. Though Ballard was not a political writer in a narrow party sense, it can certainly be read as a premonition of the selfish Thatcherite society to come – a man-eat-dog society as well as a dog-eat-dog one.

High-Rise has a clearly Freudian element to its three main protagonists. Richard Wilder (played in the film by Luke Evans) represents the id; Dr Robert Laing (clearly referring to RD Laing, the author of The Divided Self, and played by Tom Hiddleston) is the ego and the building’s architect, Anthony Royal (Jeremy Irons), who lives in one of the penthouse apartments, is the super-ego. The tower block and the wider city are conceived of as living organisms, of having a consciousness of their own. “Like a huge and aggressive malefactor, the high-rise was determined to inflict every conceivable hostility upon them.” The calming lines of the rectilinear tower contrast with “the ragged skyline of the city” which “resembled the disturbed encephalograph of an unresolved mental crisis”.

Ballard was inspired by Balfron Tower in east London. Photograph: Construction Photography/Alamy

At this psychodynamic level the residents actually enjoy the breakdown of the building’s services, and the growing confrontation between the floors. But this is no class war – the residents are all middle-class professionals – it’s territorial, atavistic. Defensible Space: People and Design in the Violent City (1972) by Oscar Newman, an American professor of urban planning, was a big influence on Ballard. Newman, like Jane Jacobs before him in The Death and Life of Great American Cities (1961), argued that urban violence can be mitigated by designs and layouts that exploit the natural surveillance of open spaces inside and outside buildings, something that high-rise buildings notably lack.

There were many architectural inspirations for High-Rise – Le Corbusier’s Cité Radieuse in Marseille, the Montparnasse tower in Paris – but the closest model is probably the brutalist Balfron tower in east London, not far from where Ballard puts his cluster of five tower blocks (more or less in present-day Canary Wharf). Its architect, Erno Goldfinger, like Royal, lived in one of the penthouse flats shortly after its completion but moved back home to Hampstead after only a few months. Not only did Goldfinger end up as the sinister Royal in High-Rise, but as an arch Bond villain – his Willow Road neighbour Ian Fleming objected to his modernist house, which was supposedly out of keeping with the street’s Georgian housing.

High-Rise’s producer Jeremy Thomas had been trying to get the novel filmed for nearly 40 years. An early Paul Mayersberg script set it in the middle of the desert in Arizona, and a more recent adaptation for the director Vincenzo Natali had it located on a floating Burj Khalifa-like megatower – all of which, curiously, was to miss the central point that the building is mentally, rather than physically, cut off from the city; the structure turns its back on the metropolis by choice not circumstance.

For film-makers it’s a challenge to convey Ballardian space not only because of the technical difficulties in rendering “inner space” but also because all the author’s fiction is in a sense set in the near future – what he called “the next five minutes”. David Cronenberg’s Crash (1997) has probably managed this best, successfully relocating the Westway, a dual carriageway in west London, to an anonymous Toronto.

Encouragingly, Wheatley’s new film, scripted by Amy Jump, is set in London in the 1970s (though filmed in Bangor, County Down) when the book is set. “It’s a moment in design that looked to the future and was still excited about it,” he says. “Now we mainly see dystopia or a white, shiny iPod future. The idea of a book looking to a future that has already happened and making a film looking back to the past to show a possible future was interesting.”

JG Ballard in 2004. Photograph: Barry Lewis/In Pictures/Corbis

The screenwriter and director Bruce Robinson, best known for The Killing Fields and Withnail & I, really went to town on the Freudian view of High-Rise in his little-known 1979 script, which he subtitled An Analogy. “I wrote it from the perspective of the building itself going insane,” he says, “with the superego in the penthouses, the middle floors as the ego and the id in the underground car park. The brain of the building goes nuts. Architecturally, the thing that interested me was the pre-stressing technique with cables. As you put on each new floor, the weight and the stress on the cables gets more and more until by the final floor these cables are so stressed that the whole building has a monstrous concrete migraine – it just wants to explode anyway.”

Reading Robinson’s script, it’s interesting to see how the writer and director of Withnail & I, with its brilliantly manic vernacular, has dealt with Ballard’s rather measured, abstract tone. “Ballard was such an innovative and interesting writer but his prose style wasn’t something I loved,” explains Robinson. “But High-Rise is an amazing piece of work, an extraordinary story.”

In both Withnail & I and High-Rise the characters must adapt to their harsh new surroundings. When the two main characters of Withnail go “on holiday by mistake”, they arrive at Uncle Monty’s cottage in the Lake District and have to improvise cooking a chicken without a roasting tray, propping it up astride a brick. There are moments of culinary ingenuity in Robinson’s High-Rise, too, when Laing (here renamed Lovall) cooks his bacon by ironing it, and the guests at one of the decadent penthouse parties tuck into seagull and gin.

These cloistered, self-enclosed environments such as the high-rise building were taken up again at the end of Ballard’s career in another tetralogy of books: Cocaine Nights (1996), Super-Cannes (2000), Millennium People (2003) and his final work of fiction, Kingdom Come (2006). Here, Ballard looked at gated communities and the “nerve tonic of violence” that he claimed was needed in order to shock his characters out of the boredom brought about by consumer capitalism, where our most difficult moral decisions involve choosing which colour kettle to have. These leisure complexes, business parks and shopping malls were now not just self-enclosed but often fortified, too.

Ballard argued that “people aren’t moving into gated communities simply to avoid muggers and housebreakers – they’re moving in … to get away from other people. Even people like themselves.” In this way, Ballardian environments actively select for psychopathic traits and it’s the egocentric Laing who is best adapted to the high-rise who ultimately survives all the tower can throw at him. At the end of the novel he finds contentment as all the lights go out in another of the five towers, “ready to welcome them to their new world”.

• The UK premiere of High-Rise will be on 9 October as part of the BFI London film festival. bfi.org.uk