Faust is always with us. When it first appears in written form, in 1587, much of the narrative is already here: the signing of a contract in blood, a period of worldly power, including great influence over specific royal courts and the ability to call great figures back to life. Other elements in the story, such as a woman whom Faust drags down and who ultimately redeems herself, and sometimes, him too, would have to wait.

But the story goes back long before that. Scholars trace it to Simon Magus in the Acts of the Apostles, the magician of Samaria who tries to purchase the holy powers of the Apostles. Some form of the bargain that makes a gain at too great a cost surfaces in imaginative literature throughout history – Great Expectations and The Godfather are variations on the Faust story. It may be one of those fundamental narratives, like Cinderella, that seems to be hardwired into the human brain to help us make sense of a sequence of events. Tony Blair's story was frequently described in a casual way as "Shakespearean", but "Marlovian" would have been more accurate. The tragedy of a politician who is seen as selling his principles for power fits the Faust plot with uncanny precision. If Blair looked like Faust, a small investigation into how the unusual adjective "Mephistophelean" followed Peter Mandelson around is rather instructive.

The great period of the Faust story was probably from the end of the 18th century to a moment just after the second world war, marked by two titans of German literature. Goethe produced a grand metaphysical exploration of social idealism, fantasy, knowledge and redemption in the two parts of Faust, published in 1808 and 1832. (He had been thinking about it as early as 1772). Thomas Mann's Doctor Faustus, first published in 1947, used the story to reflect on the consequences of German spiritual hunger in the previous half-century.

But the power of the story can not be limited to German culture. Opening this week at Covent Garden is a production of a famous and derided Faust, the 1859 opera by French composer Gounod. It was, for a long period, the most colossally popular of all operas. For decades in the late 19th century, it would open the season at New York's Metropolitan opera. It surfaces in such unlikely bedfellows as Mann's The Magic Mountain, where an early record of Valentin's Prayer summons Joachim from the dead, and the Tintin adventure The Castafiore Emerald, whose heroine never seems to sing anything but Gounod's Jewel Song from Act three. Both Mann and Hergé knew that their readers would recognise these highlights.

In recent decades, it has fallen out of fashion, and it would be a mark of ingenious perversity in intellectual circles to express admiration for it, like enjoying the painting of Mucha or the novels of Ouida. Its musical means seem quite tame to modern audiences; its visions of damnation (a tamtam) and of heaven (harps) have, at best, a delicate period charm. Gounod, on the evidence of his famous Ave Maria, was the sort of composer who thought that what a Bach prelude really needed was a big tune on top. Still, his opera goes on being performed to presumably unironic appreciation. According to the exhaustive opera database operabase.com, it is currently the 34th most frequently performed opera in the world, beating Parsifal, for instance, into a cocked hat.

After Goethe, musical versions flooded onto the stage and into the concert hall. Both Beethoven and Schubert wrote small-scale settings of some of his beautiful lyrics - Beet-hoven set Mephistopheles' song of the flea as early as 1809. Some versions stick much more closely to Goethe than others. Compare Schumann's Scenes from Goethe's Faust with Berlioz's Damnation of Faust. Schumann, setting large stretches of Goethe, accounts for Faust's death with a lovely long aria in which Faust envisages the draining of a swamp and embarking on a programme of social improvement, as Goethe does. Berlioz's Faust is tricked into going for a ride, kicking some nuns in the face and ultimately falling into an abyss where his soul is subjected to appalling tortures. (Mephistopheles: "Je suis vainqueur!").

Although attempts to stage it have been made, Goethe's text was not intended for performance, but for reading at home and the exercise of the imagination. Other versions, too, have moved towards the intimate rather than the grand. Berlioz's concert version has often been staged, by luminaries such as Terry Gilliam for English National Opera, but it is full of touches for the mind's eye only – swift cutting between scenes, huge landscape shots described as "Another part of the plain. An army advancing" and following a character in quick motion. Berlioz's demands would ultimately be achievable only on film. Other versions, such as Liszt's Faust symphony or Wagner's Faust overture, are instrumental meditations on ideas, or thematic principles; Schumann's concentrates on moments of lyric inwardness.

But there are serious attempts to adapt it, to some degree for the operatic stage beyond Gounod. Verdi's librettist Boito wrote a fascinating opera of his own, Mefistofele; Busoni, in the most musically ravishing of all stage versions, composed a problematic masterpiece drawing on puppet techniques, Doktor Faustus. There is, too, a very strong Faust element in the 1951 Stravinsky/Auden Rake's Progress, which adds a Faustian contract-signing and Mephistopheles figure to its purported source in Hogarth.

Since then, Fausts have thinned out, but not disappeared; Schnittke's richly enjoyable and fabulously tawdry 1982 cantata, Seid Nuchtern und Wachet, full of doomy tangos, is a rare Faust exercise of its time. Covent Garden is accompanying its Gounod production with two commissioned operas on the Faust theme in the Linbury theatre by Luke Bedford and Matthew Herbert. They must be the first versions of it for a while. But even apart from these explicit settings of Faust stories, it has a knack of creeping in elsewhere. When, in Strauss's 1919 Die Frau ohne Schatten, the Dyer's Wife signs away her shadow and her fertility, the unthinking allusion is clear.

Why, for a century and a half, did the story exert such power over composers? To some extent, the answer is to do with the stage. Just as the technology of stage spectacle was venturing onto a new scale of boldness, it offers hellish flames, magical appearances and disappearances and any number of opportunities for striking lighting effects. (Striking orchestral effects, too, like Berlioz's music for the ballet of the sylphs and the howling trombones in the ride to the abyss). It is also to do with the way the story conforms with operatic conventions - the three central figures, Faust, Mephistopheles and Gretchen fall into the established trio of tenor hero, bass-baritone confidant/villain and soprano heroine. Gounod, Boito (after some persuasion) and Stravinsky follow this obvious path. Schumann, high-mindedly, makes Faust a thoughtful baritone. Busoni with characteristic perversity omits the Gretchen altogether, and makes the hero an intellectual baritone, Mephistopheles the tenor at the footlights, rather like the two central figures in Schoenberg's Faust-influenced Moses und Aron.

But there is more to it than that. What opera needs for its central figures is a heroic variety, and the Faust story supplies this very readily. Each of the main players has such different aspects that a composer isn't limited to one sort of music. Bass baritones adore singing Mephistopheles, not just because villains are such fun, but because his part usually contains such various high points as Berlioz's exquisite Voici des roses and the final ride to the abyss. Both Faust and Gretchen have very different sides to their characters, and offer rich possibilities to both singer and composer.

Will Faust ever disappear? It seems unlikely. The spectacle that thrilled a previous age will fade, but its fundamental question will always mean something. What would you do to attain the thing you dream of?