I admit it. I used to think Elgar was a composer who could be encapsulated by the clichés of the Last Night of the Proms and his handlebar moustache, that he and his music were mired in a British backwater of fusty romanticism that was outdated even in its own time, and doomed forever only to speak of ludicrous Imperial ambition and patriotic stiff upper lips.

Chromolithograph - published 1912 - of Edward Elgar. Photo: Universal History Archive/Getty Images.

The piece that first opened my ears to the idiocy of my thinking about Elgar was the Second Symphony, especially its slow movement. This music glows with a strange, veiled radiance that is one of the most special sounds a late-romantic composer ever conjured from the orchestra. There's a particular passage of unsettling visionary power (you hear it twice in the movement) in which Elgar simultaneously dissolves and recomposes his orchestra. A long-breathed melody happens somewhere in the strings and woodwinds - even in the score it's difficult to see precisely how and where the tune is being played, such is the richness of Elgar's orchestral writing - but surrounding it is a gossamer tracery of harp lines and of divided violin, viola, and cello parts that glitter and shimmer. The noble outline of the melody is transformed into a much more ambiguous dream-state by an astonishing feat of orchestral imagination, in which colour and timbre become a way of feeling. It's as sensuous a soundworld as Wagner found in Parsifal, it's as precisely heard and ethereally effective as anything in Debussy. But its expressive combination of tenderness and melancholy is something that belongs definitively to Elgar.

Elgar the colouristic genius: that's just one possible way of hearing one part of the Second Symphony. There are many more: the whole piece is staged as contrast between an outer world of large-scale symphonic structure and an inner, private realm of dream, doubt, and sometimes nightmare. Elgar's unique mastery in this huge, hour-long piece is that he creates enough symphonic momentum to carry you through the music's architecture, while at the same time undermining that monumentality with moments of visionary stasis, lyricism, and, occasionally, violence. And you feel the expressive impact of these remarkable passages precisely because they come within the framework of this gigantic symphonic design. My point is this: Elgar's Second Symphony isn't just a strikingly modern phemomenon for 1911, when Elgar conducted its world premiere at the Queen's Hall, which belongs right up there with the greatest pre-first world war symphonies that were being written anywhere in Europe. The Second Symphony dared things that are still strikingly contemporary and achingly moving.

Elgar second symphony's dedication to King Edward VII Photograph: theguardian.com

A heady mix of influences in Elgar's private and public lives came together in this symphony. There was his relationship with Alice Stuart-Wortley, his muse and inspiration around this time, to whom he wrote that he had "shewn my soul" in this music; parts of the symphony were written in Venice, whose St Mark's may have inspired the opening of the slow movement and the scherzo; the piece is dedicated to the memory of Edward VII, who had died in 1910, and the slow movement is often thought of as a monarch-memorialising funeral march, but the music is surely also inspired by close friends who had died in recent years; and Elgar appended a quotation from a Shelley poem – Rarely, rarely comest thou, spirit of delight! – to the score.

That sense of struggle, of nagging doubt below the apparently confident surface, of that elusive attempt to capture the spirit of delight, is there right at the very beginning of the symphony, when the opening note has to wrench itself into being in the strings and woodwinds for a few tortured seconds before it blossoms into the first melody. Even more clearly, you hear it at the centre of the movement, which is becalmed in a strange stasis of limpid but extraordinarily detailed colours – listen to the way the melodies are shared between the strings, and surrounded by a weird chromatic halo of harps and woodwind writing – before the main tune returns, with a spirit of somewhat forced triumph.

The scherzo contains the symphony's nightmare, a crescendo that inundates the orchestra with a huge wave of rhythmic power. It's a moment of bewildering violence in the midst of the scherzo's otherwise filigree virtuosity. And it represents, Elgar said, "the madness that attends the excess or abuse of passion" – a telling phrase in the context of Elgar's life at the time – and linked it to a horrifying image in a Tennyson poem that imagines a conscious corpse in the grave being run over by horses whose hoofs, "beat, beat into my scalp and brain".

At the premiere, having got to the end of the deceptively serene finale (the symphony closes with a radiant quietness, the opposite of the tumult of triumph that had capped his First Symphony) Elgar was bemused by the less-than rapturous reaction of his audience. He asked the orchestra's leader, WH Reed, "What's the matter with them, Billy? They sit there like a lot of stuffed pigs". Rather than the porcinity of the public, I think that incomprehension is proof instead of the symphony's enormous expressive richness and complexity. This piece is essential Elgar because its rhetoric is the exact opposite of obvious emotionalism or taken-for-granted symphonic grandiosity. It's ambiguous, elusive, multi-layered, which is why it's so emotionally powerful – and why it's one of the 20th century's most important symphonies.

Five key recordings

Edward Elgar/London Symphony Orchestra: the original recording and still exemplary - there's still more desperation and surging energy in Elgar's own recording than any that have come after. And you can hear part of the rehearsals from the sessions in 1927here.

Colin Davis/London Symphony Orchestra: huge intensity and lyricism from Davis and the LSO recorded live at the Barbican in 2002.

Mark Elder/The Hallé: Elder and his Manchester players prove why they've created a new performance practice for Elgar - and Sir Mark recites Shelley, too.

Giuseppe Sinopoli/Philharmonia Orchestra - Sinopoli's controversially slow performance reveals the modernity of the colours and counterpoints of Elgar's symphony.

Sakari Oramo/Royal Stockholm Philharmonic Orchestra - Oramo makes Elgar his own in his recent recording, and claims a place for the piece in wider European repertoires.