Several elements were vital to the effectiveness of the 1979 horror film Alien, which was essentially an old-fashioned haunted house story relocated to deep space. (Its own director, Ridley Scott, called it "a C-movie done in an A-way".) Chief among them was the visceral and disquieting design work by the Swiss surrealist artist HR Giger, who has died aged 74 from injuries sustained in a fall.

Giger's "biomechanical" style was born out of his experience of night terrors and the art therapy in which he partook to combat this sleeping disorder. It is fair to say that he has been responsible in his own way for disrupting the sleep of others. "People are either thrilled or terrified by Giger's art," said the Austrian artist Ernst Fuchs. "No one else knows how to depict the most horrific nightmares so stunningly beautifully." The novelist and film-maker Clive Barker observed: "Giger seems to be painting aliens but the closer you look, the more you realise he's painting twisted versions of us."

Alien centred on an intergalactic cargo vessel which touches down on a desolate planet in response to a distress signal. The crew inadvertently picks up in the process a carnivorous life form. It later bursts gruesomely from the chest of one crew member in the most memorable entrance of any movie character since Orson Welles stepped from the shadows in The Third Man. The infant monster is smooth, eyeless and bulbous, both strongly foetal and absurdly phallic, with a row of silver milk-teeth and a lashing, segmented tail. "It was Francis Bacon's work that gave me the inspiration," Giger said. "[It] would come tearing out of the man's flesh with its gaping mouth, grasping and with an explosion of teeth … it's pure Bacon."

The alien flees the scene of its birth and is glimpsed at subsequent stages of its accelerated development as it picks off the crew one by one. Still apparently without eyes, it has now grown as tall as a Harlem Globetrotter. Its entire head takes the form of a gleaming, elongated shell that suggests a futuristic crash helmet. Within its vast jaw are rows of teeth emerging like drawers in a filing cabinet. A tendency to drool lends it a lascivious element. All this grotesqueness never quite undermines its allure.

In the final scene, the monster is blasted into space by the only survivor, Ripley, played by Sigourney Weaver. Both she and her nemesis returned in three sequels of markedly contrasting flavours: James Cameron's wham-bam Aliens (1986), David Fincher's clammy, intense Alien³ (1992) and Jean-Pierre Jeunet's comic-book-style Alien Resurrection (1997). Giger's designs were central to each of those sequels, as well as two crossovers with the Predator franchise – Alien vs Predator (2004) and Alien vs Predator: Requiem (2007).

But his involvement was not always harmonious, or even acknowledged. "With the fourth Alien film, they just took my creations, they used my 'chest-burster' and they didn't even give me any credit. It's offensive." He had a happier experience contributing to Scott's own Alien prequel, Prometheus (2012).

Giger was brought on board Alien at the suggestion of its screenwriter, Dan O'Bannon. Both men had been collaborating in the late 1970s with the cult director Alejandro Jodorowsky on an adaptation of Frank Herbert's science-fiction epic Dune, which was never made (though Giger's designs for the abandoned project can be seen in a 2013 documentary called Jodorowsky's Dune). O'Bannon introduced Giger's 1977 book Necronomicon to Scott, who seized in particular upon the painting Necronom IV, and commissioned him to design a creature based on this.

"I was the first one to go see him in Switzerland, and persuade him to get on a plane," said Scott. "He wouldn't get on a plane, because he was afraid of flying. And he finally came to Shepperton. He was with me for 11 months. Never went into town, stayed over a pub in Shepperton. Very non-Giger, not exotic. You'd think he'd be in a suite in a hotel. He's in a pub. He was in a room over a pub, and he was very happy there."

The artist built a prototype incorporating Rolls-Royce parts, rib bones and reptile vertebrae. His responsibilities expanded also to include the design of a partially fossilised figure (sometimes referred to as the "space jockey") seen when the crew explore the planet, as well as the planet itself (LV-426). Plainly put, his influence permeates Alien. Giger was deservedly part of the team rewarded when the film won the Visual Effects Oscar in 1980.

He was born Hans Rudolf Giger in Chur, Switzerland, which he called "unbearable", characterised by "high mountains [and] bourgeois attitudes". The family home was a place of early and inspiring terror. He later wrote in Necronomicon of the cellar as "a monstrous labyrinth where all kinds of dangers lay in wait for me" and of "steep and treacherous wooden stairways without banisters [that] led down into the yawning abyss".

Other boys played with toy cars but Giger could usually be seen dragging a skull on wheels behind him; he constructed ghost trains in the back garden. His father, Hans, was a chemist who encouraged Giger to study industrial design, which he did along with architecture at the School of Applied Arts in Zurich. His mother, Melly, to whom he was close throughout his life, was more encouraging of his provocative style of painting, drawing and sculpture. An early muse was the actor Li Tobler, with whom Giger had a tempestuous relationship. Tobler, who killed herself in 1975, was the inspiration for the wan, wilted females in his paintings.

Giger worked predominantly in inks and oils at first. His use of the airbrush soon became integral to his art, bringing a slick smoothness to images which oscillated between the grisly and the sensuous, often accommodating both. He prized the airbrush's "tremendous directness" and said that it enabled him to "project my visions directly onto the pictorial surface, freezing them immediately". But he abandoned it near the end of his career when it was adopted by artists with whom he did not want to be associated: "I could damage my reputation, since much of what they do is pure kitsch. I keep myself apart from that. I see myself as a surrealist."

He gained widespread exposure after being featured on the cover of the 1973 Emerson, Lake and Palmer album Brain Salad Surgery. In the early 70s, he made several short documentaries about his work. His fame increased following the release of Alien, and he took on occasional and usually unfulfilling work on other films, among them Poltergeist II: The Other Side (1986), the Alien-influenced Species (1995) and the 1996 German horror-comedy Killer Condom (tagline: "The rubber that rubs you out!").

He also collaborated on several Giger bars, including two in Switzerland, which reproduced his aesthetic in a social setting.

His first marriage ended in divorce in 1982. He is survived by his second wife, Carmen, director of the HR Giger Museum in Gruyères, Switzerland, whom he married in 2006.

• Hans Rudolf Giger, artist and set designer, born 5 February 1940; died 12 May 2014