"Don Giovanni is called the director's graveyard," says Kasper Holten. "It's impossible to do a perfect production. The existential moral journey of the seducer to hell is hard enough to make convincing – without having to juggle all the farcical elements, too."

So why is Holten, the Royal Opera House's director of opera, returning to Mozart's work for the third time (he has already directed it on stage and on film)? And why is he ratcheting up the risk with some of the tricksiest, most perilous video design ever seen on the British opera stage?

"It makes sense marrying video technology and Mozart," he explains. "If he were alive, that's what he would want on stage." And what is on stage is a rotating, Escher-like palazzo, dense with improbable staircases and doors. The set rotates on an axis, opening and closing like a doll's house. And, as if that weren't enough of a logistical challenge, video images are projected on to this rotating contraption – a superb technical feat thanks to software developed by D3, a British firm based in south London.

In the opening scene, Giovanni's servant Leporello climbs a ladder and starts writing a name on the balcony wall. The digitised projection that follows his hand was culled from a royal calligrapher, video designer Luke Halls tells me. The calligrapher's writing that has been digitised is then projected as emerging handwriting at the right moment. But that's nothing. Next, the whole facade begins to fill with names until, by the end of the overture, the hundreds – no, thousands – of women seduced by Giovanni have been recorded. Respect to Halls, apart from anything else: he's reviving the dying art of handwriting.

Halls's video projections are visually intriguing and sometimes brilliantly illuminating, nowhere better than when Elvira sings of her betrayal and a projection of fluttering bird silhouettes ornament her sorrow. Although at first a bittersweet counterpoint, it rapidly becomes something much more threatening as the beating wings of a bird of prey emerge and dominate, evoking both the predatory Don and the vengeful ghost of the Commendatore. If, at other points, they seem a tad obvious (at the death of the Commendatore, the palazzo facade fills with a wash of red), what's undeniable is that the video is ingeniously integrated into the director's artistic vision.

Hell, for him, is being alone... Mariusz Kwiecien as Don Giovanni. Photograph: Tristram Kenton

Holten's big idea is that Jean-Paul Sartre was wrong: hell isn't other people. "Hell, for Don Giovanni, is being alone. Today, we're not afraid of God's wrath – we're more afraid of there being no God."

In Halls's video projections, the names projected on the walls and doors of the Don's palazzo become a signifier of his incessant need to seduce, to do anything to elude the unbearable fate of being alone. While Leporello tells of his master's many lovers, each doorway of the palazzo is darkened by a silhouette not of a discarded women, but of Don Giovanni – as if to say, every seduction is about his narcissistic psychodrama, his skirt-chasing an existentially manic displacement activity. Arguably, though, the most poignant moment comes when the video projections erase themselves from the stage as the Don gets his comeuppance. It's a risky strategy: his seductive powers are stripped bare, the braggadocio of his sexual swaggering dwindles until he cowers on the bare wood of an empty stage.

Video didn't used to be as sophisticated as this, nor as subtly integrated into the overall artistic vision. I remember my own first experience of video design in opera at the Paris Châtelet theatre in 2000 during Peter Sellars's staging of John Adams's opera-oratorio nativity story El Niño. Then the video seemed part of a confusing multimedia onslaught. On stage, there were singer and dancers, all of them duplicated on a video screen. I wasn't alone in finding it too much: when the New York Times reviewed the American premiere of the same production, under the headline "With Ears and Eyes in Fierce Competition, the Eyes Have It", the critic worried that "Mr Adams' music is reduced to a soundtrack ... [It] bears so many gifts that the senses stagger under the load."

That's the perennial worry: video design risks duplicating, distracting from, or clunkily enforcing a symbolism we already grasped from words, music and staging. Viewed thus, plumping for flashy video design risks looking like a panicky reaction to titivate a production from a director who doesn't have much of an artistic vision, but a few grand left on the production budget. "Video has changed a lot since those days," says Halls, who has worked on U2 concerts and the 2012 London Olympics closing ceremony, but has only once before worked in opera. He is passionate about video's role in its future: "But it is a heavy tool. You have to use it with discretion, otherwise you risk swamping everything."

There are other worries. "There can be huge logistical problems," says Holten. "You're using a different language for one thing. A separate video screen can neutralise the lighting design. And people used to complain – rightly – that the sound from the projector interfered with the music." A new sound-proofed box hanging from the grand tier, he says, solves that last problem. "We're lucky because Don Giovanni is about the process of manipulating reality and the creation of illusions. Video design enables that. Video is almost leading the way."

But surely megalomaniac opera directors don't like that shift in the creative balance. Didn't the video guys used to just be nerdy nobodies? "It's a definite change, but I embrace it because I love collaborating, I love seeing a production and wondering if a particular idea was mine or someone else's. The thing that's most exciting for me is finding Luke, who's not just a tech nerd, but he's got an artistic language," says Holten.

Other opera directors, particularly those with film or theatre backgrounds, are also finding collaborating with video designers fruitful. Simon McBurney even had a video artist standing stage left and projecting images on a screen behind the singers during his recent English National Opera production of Mozart's The Magic Flute. But that was only one of the elements that made McBurney's production the most technically sophisticated and, for the audience, sensorily demanding, in ENO's history. There were video backdrops, and a foley artist working in a booth stage left; the raised orchestra pit brought the musicians into view, and the singers were miked up for parts of their performance.

Determined to reinvigorate the medium … Simon McBurney's production of A Dog's Heart at the Coliseum, London. Photograph: Tristram Kenton for the Guardian

In his approach to opera, McBurney is determined to reinvigorate a medium he once found "wooden, obscure, and pointless". His debut production, Alexander Raskatov's adaptation of Soviet dissident writer Mikhail Bulgakov's A Dog's Heart in 2010, featured video projections by Finn Ross that included a snowstorm backdrop, agitprop banners, archival crowd scenes and even Muybridgesque sequences of running dogs and men towering over the singers on stage.

Monty Python's Terry Gilliam also worked with Ross on his debut for ENO, Berlioz's The Damnation of Faust in 2011. Gilliam took the French composer's interpretation of Goethe's drama and imagined the eponymous hero being lured into the gimcrack grandeur of Nazism, before winding up murdered in a gas chamber. For this, Ross screened projections of swastika banners and ranks of helmeted stormtroopers. Even those critics who got queasy ("Should the Holocaust be trivialised this way for the sake of a coup de theatre?" asked one, rhetorically) couldn't deny that the production was bursting with theatrical flair, the video design fully integrated into the director's vision – not least when Faust and Mephistopheles jumped on to a motorbike and sidecar to take a thrilling night ride among dazzling, whirling video projections.

Ross will again collaborate with Gilliam as he tackles Berlioz's Benvenuto Cellini later this year. Is opera in 2014 increasingly unthinkable without video design? "No," says Holten, "but opera has never been afraid of technology. We forget that, when we yearn for traditional opera. Opera isn't, despite what some think, conservative. It's always been married to the latest technology. When Wagner produced opera, he got some of the most exciting technological innovations at Bayreuth. We're now trying to push the artistic boundaries even further."

• The performance of Don Giovanni on Wednesday 12 February is broadcast live in cinemas across the UK and internationally. Details: roh.org.uk/cinemas