One of the most important things about Beethoven's Eighth Symphony is that it puts a definitive kibosh on the idea of a symbiotic relationship between a composer's biography and their music. In the summer of 1812 when Beethoven was obsessively working on this piece, he wrote the most infamous letter of his life, to his mysterious "Immortal Beloved". The pain-wracked and heartbreaking sentimentality of that letter, with its doomed love and self-pitying prostrations, finds absolutely no corollary in the fabric of the Eighth Symphony, which is the most ebulliently experimental symphony that Beethoven composed – and therefore, quite possibly the most ebulliently experimental symphony in the canon. (More proof needed? I give you the Heiligenstadt Testament, that despairing document in which Beethoven realises the full magnitude of his hearing loss, written around the time of his thrillingly self-confident Second Symphony.)

First performed in public at a concert in 1814 in Vienna that also included the Seventh Symphony and Wellington's Victory (that work of tub-thumping jingoism that caused Beethoven to tell one of his detractors "what I shit is better than anything you could ever think up") the F major Eighth Symphony didn't generate the kind of applause that would signify "universal delight", and "did not create a furore", according to a contemporary account. Frankly, I'm surprised that the public's reaction wasn't total bewilderment - a more than comprehensible response to one of the shortest, weirdest, but most compelling symphonies of the 19th century.

What's brilliant about the Eighth's relatively small (time) scale is that it allows Beethoven to be more structurally radical than he could dare to be on the larger canvasses of his other symphonies. In the Eighth Symphony, there are holes that are left open after the final chord, questions that remain unanswered, loose ends that are deliberately not tied up. Most obviously, the Eighth Symphony has no slow movement (Beethoven did sketch one, but he abandoned it), but instead there's an impish Allegretto scherzando that comes second, a four-minute (or less) piece that was thought to be a homage to Johann Maelzel's metronome, but which is now recognised for what it is: an unprecedented intermezzo in place of an adagio. Except it isn't an "intermezzo" in the sense of being "incidental" to the music's argument, because this piece embodies the central and paradoxical substance of this symphony: this short movement, in its rhythmic obsessions, like the repeated staccato chords in the woodwind, or the demi-semiquaver chirrups of the first theme, and the bass-line that answers it; in its extremes of dynamic, often putting a fortissimo right next to a pianissimo, its hocketing textures of interlocking orchestral lines, and its warped musical mechanisms, sounds more like a proto-Stravinskian orchestral scherzo than an early romantic orchestral movement.

And that's the paradox of this symphony. It makes you think you're listening to a light-hearted witticism, but Beethoven is in reality reforming the symphony right in front of your ears. If you hear the second movement as a musical joke, you're missing the point. Beethoven is trying to make a symphony in which textural, rhythmic, orchestral and harmonic invention take the place of the expressive intensity; so much so, that the piece can do without a conventional slow movement. That's a gigantic leap of musical imagination and compositional technique, and it means that the rest of the symphony is similarly reoriented towards this goal, so that Beethoven – in this piece perhaps more relentlessly than in any other of his symphonies – is focused on specifically musical questions that create and obey their own logic rather than any pre-existing models or forms. In that sense alone this symphony is "Haydnesque", an adjective often applied to it, but usually in a way that manages to patronise both Haydn and Beethoven, as if Haydn was only capable of comic symphonic entertainment, and as if the Eighth Symphony was somehow a lesser thing than the supposed titans that surround it – which it decidedly isn't.

The first movement begins with a gesture of closure. The first two bars of the piece ought to be the end of a symphonic argument, not its beginning, and in fact the first movement ends with the very same music, now in its proper place. Continuing this inversion of common practice, the first movement soon finds weird keys, strange silences, and odd sounds – this solo bassoon, for example, or this ambiguous pianissimo. But it's the central section and the reprise of the first theme that should knock your symphonic socks off: over a strangely foreboding ticking mechanism in the violas – an alternating octave you've just heard at the end of the first section – Beethoven inexorably screws up the harmonic tension through a halting, uncertain sequence of variations on the first bar of the symphony, separated by gigantic walls of orchestral sound. That contrast catalyses a thrilling section of orchestral counterpoint, propelled by the tortuous transformation the cellos and basses visit on the main theme; Beethoven generates massive harmonic and rhythmic friction here which is at last released in a triple fff (forte-fortissimo!) restatement of the first theme in the bassoons, cellos, and basses. It's a moment when Beethoven gets rightly carried away with what he's done: that triple forte in the rest of the orchestra is so loud that it tends to obscure the tune in the bass line.

Charles Mackerras found an excellent solution in his performance; Colin Davis and Hans Pfitzner (Pfitzner with a theatrical change of tempo) get the balance better than anyone else in my list of recordings.

The third movement is Beethoven's only symphonic minuet: a stately antipode to the Allegretto second movement, people often say, but that's again only if you choose not to hear what Beethoven's doing under the surface of the music. The piece is called only "Tempo di menuetto" - in the "time of a minuet" rather than a real courtly dance, suggesting that Beethoven is playing with instead of inhabiting the genre of the minuet. He luxuriates in a different soundworld from the rest of the symphony – sensuous and lyrical rather than crystalline – and plays with your sense of pulse and metre.

The finale starts with an existential itch: a pianissimo aggravation in the violins that sounds like a scurrying upbeat to a tune that never comes. Instead, after subsiding to piano-pianissimo, there's an orchestral onslaught, built over those alternating octaves again, which you'll hear throughout this movement, marking time but fragmenting orchestral space, especially when the timpani have them with the bassoons. Beethoven starts his second theme in A flat major, he atomises his itching idea into its constituent elements and disperses it over the orchestra; he manages to wrench the music from F sharp minor and B minor back to F major in an astonishing sleight of ear. He creates a Klangfarbenmelodie, a melody of changing orchestral colour, a century before Schoenberg and Webern had the idea, he makes silence, dramatic pauses, integral to symphonic discourse in a way the symphony had never done before, and he creates a barnstorming coda that seems out of proportion to the rest of the movement, which makes you ask: what on earth just happened? Instead of resolution, the Eighth's fundamental musical questioning goes on long after the piece has finished. And it's quite possible it doesn't have an answer – and just as well, too: keep on listening, and keep on asking those questions!

Five key recordings

Orchestra Révolutionnaire et Romantique/John Eliot Gardiner: the shock of the new in Gardiner's thrilling performance.

Chamber Orchestra of Europe/Nikolaus Harnoncourt: once radical, Harnoncourt's performance now seems to steer a middle ground between early music energy and modern instrument sumptuousness.

Staatskapelle Dresden/Colin Davis: don't let the luxuriance of the Staatskapelle sound deceive you – this is as insightful and illuminating performance as they come.

Vienna Philharmonic/Christian Thielemann: Listen to Thielemann's "adventurous conservatism", as Joachim Kaiser puts it, in this imaginative and sometimes interventionist performance.

Berlin Philharmonic Orchestra/Hans Pfitzner: the performance - from the early 1930s - that made Colin Davis want to become a conductor, and more than a fascinating historical document, a revelation of an Eighth from another world.