The high arts of literature and music stand in a curious relationship to one another, at once securely comfortable and deeply uneasy – rather like a long-term marriage. At the securely comfortable end of the emotional spectrum we have those zeniths of song, the German lieder tradition, and high opera. In the best examples of both forms words and music appear utterly and indissolubly comingled. However, at the other end of this spectrum we have those kinds of music that attempt to be literary – so-called programme music – and those forms of literature that attempt, either through descriptive representation or emulation, to aspire to the condition of music. It is not my wish to denigrate works of these type, nevertheless there does seem to me to be an inevitable compromise – deterioration even – when an art form, rather than proceeding entirely sui generis, finds its ground in another form's practice.

I myself am a latecomer to the serious appreciation of serious music – apart from jazz, which in the hands of practitioners such as John Coltrane or Thelonious Monk rises to the inventive musicianship and self-enclosed expressiveness of the greatest that small-ensemble classical music has to offer. Still, there comes a point in everyone's life when it's time to largely put away such childish things as electric guitars and harmonicas, and it may be precisely because I was in my 40s when I began to really hear symphonic music that I have approached the form altogether untrammelled by received ideas about it – a fancy way of admitting complete ignorance. There's this, and there was also an intuition I had that my own practice as a novelist – when, that is, my mojo was properly working – had far more in common with how composers conceive of the symphonic, than it did with the lit-crit – let alone the "creative writing" – view of how it is writers actually write.

The search for motifs, or themes, the creation of an alternative world in words, the struggle for authenticity of narrative voice, the counterpointing of different protagonists' views – these are key artistic objectives shared by the novelist and the symphonist, and not to anything like the same degree by other musical and literary practitioners. Indeed, I'd go further: the symphonist and the novelist have more in common with each other than they do with others working in their own respective art forms. Why this isn't widely recognised, is, I think, a function of the essentialist fallacy that expects words-about-music to do the same thing as music alone, and music-about-words to do the same thing as words alone.

By any literary standard, Strauss's Till Eulenspiegel or his Don Juan do little in the way of hitting the narrative and characterisation buttons – nor do these tone poems succeed in accurately representing the worlds they aim to depict in the way that even a bad novel might do. To be fair, such hybridisation as Anthony Burgess enacted in his Napoleon Symphony: A Novel in Four Movements is conversely musically unsatisfying – while as literature, it's near-unreadable. On the whole novelists, rather than – as Burgess rather heroically attempted – recreating the structure of the classical symphonic form, usually confine themselves to describing the impact of music on the individual or collective psyche. This, it strikes me, is also a blind alley: for every reader who finds EM Forster's Albert Hall concert scene in Howard's End an incisive portrayal of minds in the sway of music, there's another who feels it utterly misses the point. While Proust's invention of Vinteuil's sonata, "the little phrase" of which so transfixes Swann in Remembrance of Things Past, may be effective as a literary trope, its persistent recurrence only ever summoned up in this reader a petulant desire actually to hear what the bloody thing sounded like.

No, I think it is in the realm of pure praxis that the forms really speak to one another, and to understand this we have to look at their close parallel development. The symphony owes its origins to the opera overture, which was then calved off – as it were – and inseminated by the already-mature sonata form so that it acquired its morphology of three and then four interrelated movements. This process took place – non-coincidentally, in my view – at exactly the same time the novel was in its inchoate form. However, while I don't see any necessary correspondence, say, between the symphonies of Stamitz or Gossec, and the novels of Aphra Behn or Samuel Richardson, there is a practical affinity: during the late 18th century, just as the symphony orchestra had no settled constitution, so the epistolatory novel was in the process of establishing what might be termed a unity of narrative voice as well as an effective chapter-based structure.

That both forms reach their apogee in the 19th century – and in very similar ways – seems to me a function of their sharing the same artistic aim: to simultaneously enact the most complete possible world-in-words (or world-in-notes), while also actualising the creative personality itself. For the 19th-century symphonist, the sonic cosmos he created needed to be internally consistent, while at the same time expressing his unique spirit – functions undertaken, respectively, by harmony and melody. In the great 19th-century realist novels similar aims find their outlet in the assumed equivalence of the writer with the impersonal narrative voice. This sleight-of-mind induces in the reader a conviction of the authenticity of the events described and the sincerity of the describer – harmony and melody again.

At the twin peaks of the 19th-century novel and symphony there is an overarching confidence about what the forms can do, a sense of their totalising capability. In the symphonies of Beethoven and Brahms, or the novels of Tolstoy and George Eliot, there is little insecurity about the potential of their form – no neuroticism, no insinuating irony. God remains relatively securely in his world, and the novelist or composer remains equally secure in his or her ability to interact with it in the service of producing aesthetic effects. Of course, there's trouble on the horizon – how could there not be? – but for now the enlightenment conception of progress stays in lockstep with the advance of both art forms.

The disconcerting tritone – an interval of three whole notes between tones – which Alex Ross in his magisterial history of 20th-century classical music The Rest Is Noise identifies as the clarion call of dissonant modernity, has its literary equivalent in the uneasiness that begins to infiltrate the characterisations of, for example, Henry James or Marcel Proust, and the structural devices of Joseph Conrad or Gustav Flaubert. Psychological realism, inflected by the sexual depth charge of Freudianism, is about to kill the reliable old narrator stone dead, while the great buttery sound-swaths that the other Gustav, Mahler, is plastering on to his gargantuan symphonies disconcertingly suggest in their very sweetness that you shouldn't altogether believe they are butter.

Mahler, whose soundscape already includes generous soupçons of quotation and allusion – postmodernism, if you will – also prefigures the formal dissolution of the symphonic form. His oft-trumpeted personal preoccupation with "resignation" and death itself is at once a quest for the plot that has definitively been lost, and an acute awareness that après him comes a dissonant deluge. Some may bemoan that vandal Schoenberg and his 12-tone flummery, but I find a straightforward honesty in the response of classical music to the modernist moment. Schoenberg himself attempted only one symphony, and that is scored for a chamber orchestra the composition of which Gossec would have been familiar with. Elsewhere, on the fringes of the musical foment, the response of symphonists is either to become self-consciously recherché – the folkloric romanticism of Sibelius or Dvorak – or, like the 20th century's most prolific symphonist, Shostakovich, to fashion a paradoxical postmodern authenticity, in which the artist is seen to be at once far more and much less than the sum of his borrowings. However, for the most part, by the middle of the 20th century the symphony has been comprehensively abandoned by serious composers in favour of forms that don't demand a search for organic unity where they no longer believe any to exist.

If only the same could be said for the novel! Certainly, western literature had its own sustained modernist moment, but while Virginia Woolf, James Joyce and others may have responded with fidelity to the death of the old gods by fashioning a prose fiction that dealt with the phenomenon of individual consciousness in a chaotic world, it didn't catch on. Be that as it may, in my view Ulysses stands at the very point where the novel and the symphonic forms approach nearest to one another. Steeped in music himself, Joyce saturated his magnum opus in all the effects of a great symphonist – its prose, like music, happens in a continuous present; he deploys colour as a modal effect with unrivalled consistency; the rhythm of his punctuation is integral to the meaning of his sentences rather than a bothersome adjunct; and perhaps most significantly of all, the entire work is conceived of as a grand exercise in the contrapuntal, as the psyches of Leopold Bloom and Stephen Dedalus call and respond to one another.

To read Molly Bloom's great gush of resigned affirmation with which Ulysses ends and then set it beside the equally self-actualising fatalism in which the final adagio movement of Mahler's ninth symphony (marked on the score "very slowly and held back") culminates, is to feel yourself in the presence of artistic twins whose birth is separated by only a few years. However, while concertgoers still crowd out auditoria to listen raptly to one musical twin, hardly anyone makes it through Ulysses nowadays. Instead, novelists have fulfilled their readers' desire for the old cosy certainties by turning their backs on the experimental truth and taking refuge in the apparent harmony of the past.

One of the bestselling literary novels of last year, Jonathan Franzen's Freedom, self-consciously models itself on Tolstoy's Anna Karenina and toddles realistically along like modernism never happened. It's as if a contemporary composer were to rescore the Eroica, making the melodies more saccharine and the harmonies more schmaltzy, then premiere it at the last night of the Proms to rapturous applause from the musical cognoscenti. To complete my trope at the outset: for a century or so the symphony and the novel made love to each other, quite beautifully. But now its artistic partner has died, the novel, instead of moving on, lies there in the dark summoning up past pleasures while playing with itself in a masturbatory orgy of populism.

Will Self will be talking at greater length about music and modernism as part of the Notes & Letters festival at Kings Place at 5pm on Saturday 8 October.