In Invasion of the Space Invaders, Martin Amis's 1982 treatise on the emergent video game medium, the British author wrote: "The video game tells a story. The better you get, the longer the story lasts. And we all know how children feel about stories." In the early 1980s, video-game stories were laughably straightforward: the aliens die in Space Invaders, the dots are eaten in Pac-Man, the ball is batted in Pong. And yet the compulsion for players to see what happened next – and, indeed, to ensure that the story would never end – was irresistible. It's proven to be an infinite saga played out by a cast of millions. As Steven Spielberg wrote in the foreword of Amis's book: "The world can never be the same again."

These foundational stories of fight or flight soon grew into more complex narratives. Technological advances allowed game directors to adopt many of cinema's techniques, and the medium matched its elder, more esteemed relative in terms of commercial clout. By the early 90s, Nintendo's profits exceeded that of all the American movie studies combined. Its profit-making power has only grown in subsequent years. In 2013, Grand Theft Auto V made $1bn within its first week of release, prompting Bloomberg to publish an article pondering whether its frenzied success might signal the end of Hollywood.

But cultural respect for the form has been slow to catch up. Video games are inherently playful, which means many non-players still believe they are also frivolous and childish. There has been an increasing effort by some game-makers to address more substantial topics and themes. Last year's Papers, Please is a bleak yet affecting study of the lives of a series of would-be immigrants who must convince the player, a border control agent for a fictional 1980s eastern block nation, to grant them passage. At one point in the game, an elderly man approaches your booth. His papers are in order but, before he passes through, he pleads that you also let his wife into the country, even though she doesn't have the correct documentation. She is next in line. You are free to choose: uphold bureaucracy or keep the family together?

Laughably straightforward … Pac-Man. Photograph: Sinibomb Images/Alamy

Cart Life is an equally effective study of contemporary life in America on the poverty line. As you scrape a living, selling coffee or newspapers, you begin to feel the grim pain of systemic unfairness and economic failure. The sense of injustice when one character is evicted from his motel room for keeping a cat is devastating. These examples are potent, but it remains rare that a video game's story is discussed seriously in artistic terms. For many, they still appear to offer little more than an endless parade of Michael Bay-esque set pieces: impressive, explosive, hollow.

During a panel discussion at the USC School of Cinematic Arts last year, Spielberg revealed how his appraisal of video games has cooled over the past 30-odd years – at least in terms of their storytelling power. "The second you pick up the controller, something turns off in the heart," he said. "It becomes a sport". Star Wars director George Lucas – whose company LucasArts produced seminal story-based games throughout the 90s – was more hopeful. "The big game of the next five years will be one in which you empathise strongly with the characters through a love story," he said. "That will be the Titanic of the games industry, because you have actual relationships on screen instead of people being shot."

There is strong evidence that the medium is approaching this tipping point. At this year's video game BAFTA award ceremony, the television writer Steven Moffatt, presented the award for "Story" and offered the crowd the stern assurance that "video games are an art form". (He later joked that he was "here to crawl to my new bosses".) The Writer's Guild now has a video game category.

It's not only indie games such as Papers, Please and Cart Life that are examining unusual themes. One chapter in The Last of Us, a Sony-funded multimillion dollar blockbuster which won the 2014 Writer's Guild award, explored a young love story between two 14-year-old girls. And the forthcoming Quantum Break aims to move things forward still further. Made by the Finnish studio Remedy Entertainment and funded by Microsoft, it centres on two students who gain the ability to time travel when they witness a disastrous science experiment. So far, so standard popcorn premise. Where Quantum Break may deliver something new, however, is through a TV tie-in, which will see the way in which people play the game affecting the show and vice versa: a bold merger of interactive and non-interactive storytelling.

Mimics the TV show … The Walking Dead

But if Lucas' prediction is correct, the path to more meaningful or complex stories is not without its pitfalls and detours. Rhianna Pratchett, the daughter of Terry Pratchett, author of the Discworld series of novels, is a British games writer who was employed to turn an existing synopsis for a Tomb Raider reboot into a full treatment. As with many blockbusters, Tomb Raider primarily tells its story through cutscenes, non-interactive clips. For Pratchett, the main challenge was in managing the creative tension between story and gameplay. "A game's design is constantly changing over the course of its creation," she explains. "So the script must shift, too. What's good for the gameplay might not benefit the story or the characters – and some of the folks you end up working alongside don't give a damn about story."

This lack of authorial control is what distinguishes Pratchett's work from that of her father's. "It's a little hard for him to imagine a situation where the writer isn't in charge," she says.

Dave Grossman is head writer at Telltale Games, the California-based game studio that has recently found tremendous success with its adaptation of The Walking Dead, Robert Kirkman's series of post-apocalyptic zombie graphic novels. Released in episodic form and split into seasons, The Walking Dead game's structure closely mimics that of the accompanying American TV show.

By the end of last year, more than 21m episodes had been sold, in which you must rescue and guide an eight-year-old girl through a desecrated America. For a studio that's closely focused on the business of storytelling, collaboration is more straightforward than Pratchett found working on an action game. "There's a hazy delineation where the writers are accountable for things like character and narrative, while the designers are responsible for the player experience, but really everybody is thinking about story all the time," says Grossman. "You can't just make up a good story; every scene has to be one where the player is involved in a compelling way. This means there's a lot of give and take."

Even so, authorial power shifts, of course, as soon as the player gets their hands on the game. "You are effectively sharing control with the player over pacing, the sequence of events, the actions of the principal character, or any number of other things," says Grossman. "These are tools that writers typically rely on to create tension and mood and generally to keep a story from falling over. It can be unsettling for anybody used to writing novels or screenplays."

Grossman is one of the most experienced storytellers working in video games today. His career began in 1989 with The Secret of Monkey Island, a black comedy created by LucasArts. "Video games seemed like an area ripe with storytelling possibilities," he says of his decision to enter the industry as a young writer. "I remember being delighted with the way I could use the dialogue options being offered to the player to create specific kinds of dramatic moments, and to present an inner monologue in a way that you can't really do in film or television. I still find that delightful 25 years later."

But, two and a half decades on, he also accepts that the medium is ripe with difficulties for would-be storytellers. "Sometimes you have to make sure that a line makes sense no matter what was said last in the script, and after an arbitrary amount of time has passed," he explains. "You learn to deftly reintroduce topics and avoid using too many pronouns, so as not to have to write six versions of every line."

Despite these challenges, Grossman points out, "We are now in a period where an action game will be criticised for having a weak story, where critics weight the overall experience in ways similar to movie or literary criticism."

The medium may be taking tentative steps into maturity, but there are many who believe that the obsession with creating games that look and feel like film or television is holding it back. Shigeru Miyamoto, the creator of Nintendo's mascot Super Mario and arguably the best-known video designer in the world once said: "I don't want to criticize any other designers, but I have to say that many of the people involved in this industry... are longing to make movies rather than making videogames."

It's a sentiment shared by Peter Serafinowicz, the British actor and comedian who provided voice acting for the recent fantasy game Dark Souls II. Serafinowicz was so fond of the first Dark Souls, which was released in 2011, that he wrote to the game's publisher requesting a part in this sequel.

Majula, a numinous cliff-top settlement, where the reeds tilt, the crows craw and the light is perennial sepia, is typical of Dark Souls II's mystical world. The characters here, a rag-tag bunch of nomads who speak in whispers and riddles, offer only vague pointers to the world's history. There is an absence of clunky exposition. Instead you must draw conclusions from the environment itself. This unusual approach has proved successful; many critics hailed Dark Souls as a masterpiece of design, something that the whispered storyline supports without obscuring.

"You're not wading through 15 minutes of poorly scripted sub-Tarantino dialogue and poor-taste jokes," agrees Serafinowicz. "Sometimes with video games, it can feel like you're watching a terrible movie. If I wanted to watch a bad movie there are plenty I could watch – some of which I've been in myself. Dark Souls tells its stories through the things you pick up, what's around you."

While The Walking Dead's lead writer Grossman considers the work his team is creating to be "my favourite thing going right now for game storytelling", but he is quick to agree with Serafinowicz that more explicitly game-like approaches can be effective. "Gone Home, from The Fullbright Company, does an excellent job of providing an engaging narrative through strictly exploratory means, like a strange sort of story archaeology."

Steve Gaynor, is one of the creative team behind Gone Home, which was praised by The Atlantic for its "literary storytelling". In the game, you explore an abandoned house, examining objects as you work to uncover the story of what happened to the home's occupants. "There's a keener sense of investment and immersion," says Gaynor. "We strove to have as little text as possible, leaving the player to determine what things mean."

Unlike Tomb Raider and The Walking Dead, with their broadly linear plotlines, Gone Home allows the player to experience the story in their chosen order, dependent on which rooms they visit first, although, Gaynor says, they create "narrative choke points" to stop players from "turning a corner and coming to the last chapter of the story without context". For Gaynor, the power lies in allowing players to build their own unique story. "Gone Home isn't so much about reading A then B then C," he says. "It's more about trusting the player to find everything they're interested in."

Naomi Alderman is a young British novelist who has also written video-game scripts for Penguin and the BBC. "Linear storylines feels strong," she says. "You can control what you're doing and you know you can provide a satisfying experience for the player. With non-linear fiction you have many more variables, it feels exciting and the player can become highly engaged, but it's challenging to make each branch as satisfying as the others."

For writers and storytellers interested in working in the video- game medium, however, choosing between these two approaches is not the most complicated problem they face. "It's not as simple as picking up a pen and writing a novel or a digital recorder and filming something with your friends," says Gaynor. "There's still a huge amount of technical know-how needed. It's a significant barrier to entry for any storyteller who wants to put their plot in interactive form."

For Alderman, a writer who doesn't have that technical knowledge, it can be difficult to find potential partners. "The majority of people who make video games don't understand or care about storytelling," she says. "So it can be hard to find collaborators."

Literary … Gone Home.

The power and importance of story in a video game is increasing. The Writer's Guild of America now has an award category for outstanding achievement in videogame writing, as does BAFTA (The Walking Dead won 2013's award for best story). But few would deny that this art sits, not alongside film and literature, but somewhere else entirely. Nowhere else does the author and reader collaborate in the telling of a story.

During last year's panel, George Lucas was dismissive of gaming's ability to have the same impact in fiction as cinema and literature. "Telling a story is a complicated process," he said. "You're leading the audience along. You are showing them things. Giving them insights. It's a complicated construct and very carefully put together. If you just let everybody go in and do whatever they want, then it's not a story any more. It's simply a game."

And yet, game makers rarely allow the player to "go in and do whatever they want". Their art is in setting the parameters in which the player can experience a story, the placement of props, the lines of dialogue, the pacing – all the familiar disciplines of the storyteller. To dismiss this authorial work is disingenuous, and perhaps dismissive of the millions of players who have been gripped by tales in which they feature as the protagonist, and have thereby lived a story first hand.