In the pale sunshine of a winter’s day, you ride along a trail in the shadow of pristine mountains. There are other outlaws with you, chatting as they ride, guiding their horses around frosted trees. You can see your horse’s muscles moving beneath its flanks, hear its grunts as you veer off the track and push it through the snow, leaving deep gouges in the untainted white. One of your number breaks into song and you choose whether to join in. Eventually, arriving at a ridge, the gang leader pulls up his horse and you slow to a trot as railroad tracks come into view below. In the distance, you can see the train you are about to rob.

If you still think of Pac-Man or Space Invaders when you hear the words “video game”, take a look at Rockstar Games’ Red Dead Redemption 2 and you will see just how far they have come. A turn-of-the-century tale of the US frontier, it casts the player as a gruff outlaw riding with a band of other miscreants, living off the land and trying to outrun the advance of modernity. In classic western style, the dream is to make enough money from robberies and other jobs to give everyone in the gang a fresh start somewhere new. But lawmen and enemies keep chasing players across an astonishing recreation of the old west, from the mountains to the plains to the hot, soupy air of the bayou.

Even something as simple as a lifelike tree takes several people months to make

The game’s virtual world is extraordinarily detailed. As a player, you can see rainwater dripping from the trees after the sun breaks through clouds and hear deer moving softly through the woods. Its characters are modelled so exactly on real people that it looks like a film. You can call out to anyone you pass on the road or in a dusty town and they will look up from a newspaper or turn towards you to respond. Its simulation is so detailed that, at first, it is difficult to believe.

Like the fictional proprietors of Westworld (the cowboy amusement park staffed by unnervingly human-like androids in HBO’s TV series), Rockstar aims to provide players with a wild west fantasy so authentic that you can forget it is not real. This realism – not just in the way the world looks, but also in how it behaves – would have been technologically out of reach only five years ago and a phenomenal amount of money and work has gone into its creation. Indeed, in the past few weeks, Rockstar has had to defend itself from controversy about its staff’s welfare after one of its co-founders, Dan Houser, said they were working “100-hour weeks”. (He later clarified that the figure applied only to himself and three other senior writers, with other developers working 42 to 46 hours in an average week, according to the studio.) The resulting game is the result of more than 1,600 people’s labour over seven years and will have cost hundreds of millions of pounds – enough to bankrupt almost any other developer.

Rockstar is in a unique position, however. Buoyed by the incomparable success of 2013’s Grand Theft Auto V, which made $1bn in its first three days and has sold 100m copies, the developer’s creative leads have the time and money to do whatever the hell they want. Speaking to some of them at Rockstar’s Edinburgh studio, Rockstar North, it is clear that they share an obsession with perfectionism and a tendency to disappear down conversational rabbit holes; I had a 10-minute chat with Alastair MacGregor, the technical audio director, about the sound of a horse breathing. The studio’s co-head, a Canadian called Rob Nelson, speaks effusively about Rockstar’s creative aims, rarely finishing a sentence before starting the next one.

‘An astonishing recreation of the old west’ ... Red Dead Redemption 2. Photograph: Rockstar Games

“What does realism mean? I think we wanted it to feel like an authentic representation of a place and a time. But how slavishly we adhere to realism, that’s a balance that we have to strike,” says Nelson. “How do you populate a world this size with enough to do? What are the things that make a city or a town feel authentic? How are people going to be hanging out in the world – and then what systems are you going to need to have them behave believably? You can’t go out in the world and have it just fall apart on you, with little robot people walking around.”

For a long time, video games were obsessed with chasing realism. To players and developers who grew up playing with pixel characters or awkward early-3D puppets on bulky TVs, the idea of a game that looked indistinguishable from real life was the holy grail. Accordingly, video-game visuals and behaviour made technological leaps every few years. The Legend of Zelda: Ocarina of Time, released by Nintendo in 1998, was the first to create a 3D world that you could freely explore. Naughty Dog’s Indiana-Jones-esque Uncharted games and their emotionally devastating The Last of Us set new standards for cinematic narrative from the mid-00s onwards, with believable characters played by talented actors. The Assassin’s Creed series, made by Ubisoft, has recreated ancient Egypt, Industrial Revolution London and Renaissance Italy. The Witcher 3: Wild Hunt, a sprawling, low-fantasy epic from 2015 in which you play a maligned monster-hunter, is the current high-water mark for video-game worlds.

But the time when pushing creative and technological boundaries went hand in hand in game development has gone. In the past 10 years or so, as the technology has started to plateau and more people have turned to smartphones or older consoles, rather than state-of-the-art PCs, games have diversified. They were never homogenous, but they are less so now than ever: look at critics’ lists of this year’s best games and you will see a pinball game about a little ant rolling a rock around an island (Yoku’s Island Express), a brief and beautiful interactive story about first love (Florence) and a game about a semi-retired god and his son battling creatures from Norse mythology (God of War). Realism is no longer the only quality by which video games measure their success. Plenty of games made by much smaller teams than Rockstar’s eschew it entirely, embracing stylistic art and outlandish concepts or exploring one fun or interesting idea in a lo-fi way. The most popular game in the world right now, Fortnite, is a cartoonish caper with no aspiration to naturalism.

‘Red Dead 2 is very different in mood from Grand Theft Auto V, although both are adult-oriented’ ... Trevor in GTA V. Photograph: Rockstar Games

Realism sets ludicrous standards, which is why most developers do not commit to it. It is tempting to think of developers as gods, conjuring a world into existence, but the reality is that it involves a tremendous amount of often tedious work. Even something as simple as a lifelike tree takes several people months to make: some to draw and model it, some to code how the wind moves its leaves, others to record and mix its rustling. Usually, realism is not worth it.

But Houser, Rockstar’s co-founder, has long been obsessed with creating games that feel as lifelike as possible. The company’s 2001 breakout hit, Grand Theft Auto 3, conjured a living, sardonic simulacrum of New York on the PlayStation 2 and since then the developer has been at the forefront of video-game realism, expanding from one studio in Scotland to nine around the world. GTA V was another landmark, going beyond a city and creating a whole county, with mountain trails to cycle around and meth-addled trailer parks to drive past.

Red Dead 2 is very different in mood from GTA V, although both are adult-oriented and 18-rated. (The first game in the Red Dead franchise was 2004’s Red Dead Revolver, followed in 2010 by Red Dead Redemption.) The latter is a black, sharp-tongued satire of the worst excesses of the US, mercilessly depicting (and, in the eyes of some critics, glorifying) the violence, megalomania, hypersexualisation and narcissism that drives its characters and its world. The former is slower, calmer, sadder and more character-driven. It is set mostly in nature and players spend a great deal of time riding around on horses. Nothing here will inspire the same controversies as GTA V’s torture scenes and strip clubs. The world of GTA V looked amazing, but the play revolved around violence: the game was to steal cars, shoot at cops and gang members and blow up gas stations. In Red Dead 2, while you can point a revolver at a carriage and rob its driver, you can also shout a greeting as you pass. You can go fishing with your friends or hunt elk in the wilderness.

‘If you still think of Pac-Man when you hear the words “video game”, take a look at Red Dead Redemption 2 and you will see just how far they have come.’ Photograph: Namco

The ability to hunt animals is one example that illustrates the work that has gone into the game. Nelson says it was important to the team that, as a cowboy, you could live off the land, hunting animals for meat and selling the pelts. That meant the animators had to create animations and models for skinning every species of animal, from deer to squirrels, and for picking up and stowing the pelts on the back of a horse. So, they took motion-captured data from real actors and adapted it, creating almost gruesomely detailed models of skinned animals. “But at some point you’ve taken the realism too far – nobody wants to watch a 15-second movie of a cowboy skinning a deer,” says Rob. “You could just keep going and going and going and you have to figure out where to draw the line.”

There is similar detail in every aspect of the game. In Saint Denis, a city modelled on New Orleans, I find a theatre where I can watch a cast of virtual actors perform a 15-minute vaudeville routine. You can pick up anything you see on shop shelves and, if you are riding a male horse, its testicles shrink in the cold. It is absurd: what player will be paying attention to animals’ testicles?

So, why does Rockstar pursue this expensive perfectionism? Is it purely because it can? “No, not at all. It’s not just because we can, it’s because we have to,” says Nelson. It is an immense version of what Disney used to refer to as the illusion of life, he says. “We’re trying to build worlds that people believe in, that they can get lost in, that is living without you, there for you when you come to it. We want as little as possible to remind you that people made it … every time that a shortcut is taken, it’s a slight reminder that it’s been made by people, it’s not real.”

Rockstar is one of the only studios left with the resources to push video-game realism as far as it can go

In a way, the job of a game coder is quite sad: it involves spending years creating very complex and impressive systems, only to make them appear invisible as possible to the player. Behind every natural-seeming moment in Red Dead 2 – someone sitting outside a saloon, a horse dipping its head to nibble on grass – are several intricate, interlocking cogs of code determining how the simulation behaves. Phil Hooker, as technical director, is the master of these systems, bringing together the work of several hundred coders.

“We wanted, in this game, to be able to interact with anybody and for them to feel like a human being,” says Hooker. “In order to do that, we need to make sure, whatever they’re doing in the world, they have the capability to react to you. So, someone sitting stirring a pot of stew needs to be able to turn to look at you briefly or react in shock if you do something more extreme, and also maybe have a conversation with you and then go back to what they were doing. The number and combinations of animations we needed to do that, and the systems we needed to put that together, were a lot more complex than ever before.

“The challenge is that, as soon as you bring something up to that level of fidelity, everything else has got to match it for the world to feel consistent and immerse you. Every now and again, you take a single step that you know is going to involve a lot of work for a lot of people and that’s where we have to be very measured.”

Slower, calmer and sadder than GTA V ... Red Dead Redemption 2. Photograph: Rockstar Games

Rockstar is one of the only studios left with the resources and bullish determination to push video-game realism, narrative and world-building as far as it can go, no matter how expensive and time-consuming it might be. For this reason, the release of one of its games is always an event. Red Dead 2 is certainly the most ambitious game yet made, by Rockstar or anyone else. Players are not able to do absolutely anything they want in its simulated world – you cannot sit down for a meal in one of Saint Denis’ fancy restaurants, much to my disappointment – but everything you can do is executed to perfection. I marvelled at the beauty of its landscapes as I played it, gave strangers a ride home after rescuing them from kidnappers, sat by a campfire and cooked the meat from a boar I hunted. (I also accidentally galloped my horse headfirst into a tree while trying to escape from bounty hunters. Sorry, Gilly.)

“The similarities with Westworld ... It’s not that it’s a western and we’re a western,” says Nelson as we talk about the obvious comparison. “It’s that we’re trying to make another world that you can go in and interact with and believe in. Just like they are.”

Red Dead Redemption 2 is out on 26 October