Launching this Friday, Deus Ex: Human Revolution could well be the most interesting mainstream video game for a decade. As with the original Deus Ex, released way back in 2000, it offers an intriguing conspiracy-led cyberpunk narrative and a structure that gives the player huge amounts of room to explore their own gameplay impulses. Through the augmentation system, lead character Adam Jensen can be gradually transmogrified into a stealth assassin or a heavily armed juggernaut, with dozens of variations inbetween.

Although hundreds of games since Deux Ex have attempted to offer this choice, few have really allowed the player's selection to make a genuine impact on the story, and on the ramifications of each mission. Human Revolution does. And as a Triple A release, it is rather lonely.

There remains an unsolved problem at the core of game design: how much freedom do players really want? Corridor shooters like Call of Duty have evolved because there is a large audience out there for very controlled, linear cinematic experiences. But at the same time, titles like Grand Theft Auto and Fallout have prospered by offering players a certain amount of autonomy in the game world. Would Black Ops have been a better game if the player had been able to run off and take side-missions in Vietnam or Cuba? Or do we need to think about it as a totally different form of entertainment?

In that respect, studios often like to hedge their bets. For example, in Elder Scrolls V: Skyrim there will be a 30-40 hour main quest, but then another 200 hours of side-missions and non-essential activities. Grand Theft Auto and Red Dead Redemption work in a similar way. The designers are supporting freedom only by dislocating it from the main narrative. They are effectively creating two interconnected, but essentially separate gaming experiences.

Providing choice within a functioning narrative is complex, because player choices often clash with pre-prepared plot sequences. I met the producers of Far Cry 2 a couple of months before the game's launch and they were burned out wrecks – they'd spent months designing the plot and its attendant cut-scenes and webs of causation, so that the player couldn't stumble upon inaccuracies or discrepancies by trying missions in the 'wrong order'. They weren't entirely successful, of course, and the nightmare of this endeavour perhaps explains why Far Cry 3 looks to be a much more focused beast.

Making sure the player doesn't experience the story in the wrong order is the least of the problems though. For a truly open world game to work, every functioning element within that environment has to be systemic and emergent. You can't have set spawn points for enemies, because what if the player creeps up on them from the wrong angle? This happened dozens of times in the first two Doom titles, which did give the player a certain amount of room to explore, and it was like shooting demons in a barrel.

So enemies have to be given systemic AI, in order that they can act within the world in a believable manner. For a while, this meant constructing finite state machines capable of reacting to a modest series of inputs: I am under attack, I have seen the player, etc. Now developers are working with more complex systems such as AI planners and hierarchical task networks so that computer characters have more advanced behaviours and can be more proactive. The likes of Killzone 3, Crysis 2 and Halo Reach all have enemies capable of working in teams, flanking the player and avoiding incoming fire. What AI characters can't do, of course, is work out where the player is in the story and communicate with them accordingly. AI characters don't do narrative improvisation. At least not yet.

Beyond all these technicalities is the fundamental concern that players don't like to get lost. In the early days of 'arcade adventure games' like Knight Lore and Dungeon Master, players were actively encouraged to map out the games themselves on graph paper, but no modern developer would ever get that idea past the marketing team (the closest we've come recently, is your ability to mark the game map in Legend of Zelda: Spirit Tracks). Shigeru Miyamoto championed the whole hub world concept to get round this, allowing players to cast off in any direction they fancied with the security of knowing they were only one step removed from a familiar locale. This set-up became a cornerstone of the platform adventure genre in the wake of Super Mario 64, but it severely limits the linear explore-ability of the world; all you're effectively doing is taking a single step in multiple directions.

What worries players most about freedom is missing out on stuff. If you find yourself in a haunted house with dozens of doors, each leading on to corridors with dozens more doors, the completist won't be happy until every avenue has been explored – that's not gaming, that's conducting a police search. For this sort of design to work, gamers need to know there are limits, and that they won't be required to memorise and revisit multiple locations. Ocarina of Time gets this just about right, providing several locations that must be re-visited with new items, but ensuring there aren't too many so that the player dreads having to backtrack over countless areas.

Deus Ex worked because it subtly reprogrammed the player's notion of freedom. It obfuscated the structure of the game behind discoverable conspiracy theories, multiple characters and a globetrotting design and it manipulated the player into perceiving character augmentation as a narrative component. In other words, the player's progress, the transmogrification of their character, became a plot point. You were the story. Or at least your thought you were; and in game design as in life, perception is reality.

Really though, what Human Revolution does is provide us with a rich functioning world and then invites us in to feel as though we're part of it, like a generous host at a successful cocktail party. But with futuristic weapons. Furthermore, as with JC Denton in Deus Ex 1, the story isn't something that's happening out there in the cut-scenes and non-player character interactions, it's happening internally: within the character of Adam Jensen. As a human-turned-cyborg who must adapt to survive he symbolises the game's post-humanist quandaries. He is the human revolution.

Still, the game has been criticised a little for its steadfast reliance on hoary old conventions like boss battles. To get past these, studios will need to start thinking about games in a new way; as totally emergent systems. In some ways, this has been happening for years. Way back in 1980 we had the seminal role-playing game, Rogue, based around procedurally generated dungeons – it kickstarted a whole genre of randomised RPGs which has most recently led to the likes of Mystery Dungeon: Shiren the Wanderer and Dwarf Fortress. In 1985, Datasoft released Alternate Reality: The City, an RPG set on an Earth invaded by Aliens; players simply had to learn how to survive, getting jobs and learning new skills. More recently, there has been the procedurally generated online co-op game Love, filled with autonomous AI tribes, and of course, Minecraft, a game based around building stuff with blocks.

There's a weird paradox here, though. As mainstream games have become more technically advanced, they have simultaneously become less structurally ambitious. This is mostly down to the huge budgets involved in triple A production and the conservatism of the marketplace: if 20 million people are happy to buy the same Call of Duty every year, where's the impetus to invest in radical new notions of player freedom and game structure? In this context Human Revolution is an astoundingly brave project.

If it is a success, maybe there will be an increased desire to give players more of a say in game stories. I'd be fascinated to see procedurally generated first-person shooters, set in worlds filled with AI characters complex enough to decide on their own factions within the environment. Whether or not a 'plot' could evolve within this set up is questionable, but of course, as Dungeons and Dragons taught us many years ago, story is a malleable commodity, and engaged players have the imagination to make up shortfalls. But what D&D also showed, with its enormous rule books, is that all good games function like good democracies: players want freedom, but they also want structure (a government) and rules (a police force). Anarchy is not a game design, it's a dangerous sand box for minority thinkers.

Certainly what's lacking at the moment is the player's imaginative investment. At Gamescom, I chaired a panel about the future of RPGs; at the start of it I asked the participants to name their favourite ever adventure title. Ray Muzyka co-founder of Bioware spoke eloquently and wistfully about the 1981 'dungeon crawl' game, Wizardry, which managed to provide an exciting and engrossing 'story' through limited graphics. He recognised the fact that what modern RPGs are doing is replacing the imagination with gorgeous visuals and epic, intricate mythologies.

Is that the only way? The future of freedom relies on developers finding a way to utilise player imagination, without sacrificing the astonishing visuals we've come to love. That may well happen when procedural graphics and complex AI combine; perhaps a publisher like EA or Ubisoft, or a developer like Valve or Bethesda will look at the success of Minecraft and see in it the kernel of a new mainstream gaming paradigm. Like many gamers I've enjoyed corridor shooters and scripted action adventures, but I'm ready for what's coming next. Those games are benevolent oligarchies. Perhaps one day soon we'll achieve a perfect game design democracy.