I was listening to Paul Simon’s Hearts and Bones album recently, for the first time in many years – the first time, really, since I was a young teenager. I bought it when it came out in 1983 and listened to it over and over. But hearing it again, and particularly listening to the title track, I was struck by a question: how did I take this back then? What did it mean to me, and why did it mean so much?

So: the title song is a beautifully worn-down response to a relationship at its end, a mix of nostalgic glimpses of happier times and a weary, bruised sense of life in the aftermath of some cathartic break-up. Listening to it as a young teenager, still a virgin and almost wholly inexperienced in such emotions, I wonder if I didn’t think this is how I want to feel. I wanted the happiness, but in a retrospective way (because then it’s done and dusted and safe); and I wanted the melancholy because it just seemed so grown-up and sophisticated and suave. I wanted, as an old joke has it, to skip the marriage and go straight to the divorce. After all – and I am hardly the first person to point this out – there is a complex sort of joy in sadness.

But can this be right? Surely what people want is to be happy. Whole philosophies (I’m looking at you, utilitarianism) rest on the premise that more happiness is always and everywhere a good thing. There is a Global Happiness Index, measuring how happy people are (Denmark tops the league). Bhutan even has a Gross National Happiness Commission, with the power to review government policy decisions and allocate resources.

It’s good to be happy sometimes, of course. Yet the strange truth is that we don’t wish to be happy all the time. If we did, more of us would be happy – it’s not as if we in the affluent West lack tools or means to gratify ourselves. Sometimes we are sad because we have cause, and sometimes we are sad because – consciously or unconsciously – we want to be. Perhaps there’s a sense in which emotional variety is better than monotony, even if the monotone is a happy one. But there’s more to it than that, I think. We value sadness in ways that make happiness look a bit simple-minded.

Sadness inspires great art in a way that grinningly eating ice cream in your underpants cannot. In his essay ‘Atrabilious Reflections upon Melancholy’ (1823), Hartley Coleridge (son of Samuel Taylor) praised melancholy as a more refined state of mind than happiness. ‘Melancholy can scarce exist in an undegraded spirit – it cannot exist in a mere animal’ is how he put it:

Melancholy is the only Muse. She is Thalia and Melpomene. She inspired Milton and Michael Angelo, and Swift and Hogarth. All men of genius are melancholy – and none more so than those whose genius is comic. Men (those I mean who are not mere animals) may be divided, according to the kind of their melancholy, into three great classes. Those who seek for the infinite, in contradistinction to the finite – those who seek for the infinite in the finite – and those who seek to degrade the finite by a comparison with the infinite. The first class comprehends philosophers and religionists; the second, poets, lovers, conquerors, misers, stockjobbers, & c.; and the third comprises satirists, comedians, jokers of all kinds, man-haters, and womanhaters, Epicures, and bon-vivants in general.

Melancholy, Coleridge is arguing, is more dignified than happiness. I suspect this is a sense that most people have – that joy is, at root, a kind of idiot pleasure, the idiom of the lobotomy, a balloon just waiting to be popped. Sorrow is somehow more grown-up, because less illusioned. It feels more sincere, more authentic. As she prepared to write Adam Bede (1859), George Eliot copied the following from Thomas Carlyle’s Life of Oliver Cromwell into her notebook: ‘The quantity of sorrow he has, does it not mean withal the quantity of sympathy he has, the quantity of faculty and victory he shall yet have? Our sorrow is the inverted image of our nobleness.’

Because it has some of the colouring of nobility, sadness is also, perhaps, more beautiful than happiness. Philip Larkin’s ‘Money’ (1973) ends:

I listen to money singing. It’s like looking down

From long French windows at a provincial town,

The slums, the canal, the churches ornate and mad

In the evening sun. It is intensely sad.

It Is Intensely Sad would be a pretty good title for a study of Larkin’s verse as a whole. Of course, one reaction to this poem would be to say: ‘Wait just a minute, Phil: you don’t actually mean “it is intensely sad”. You mean “I am intensely sad”. The street, the church, the whole provincial town is doing just fine, thank you, and has no responsibility for your mournfulness, looking down from your long French windows.’ Such a reaction would not diminish Larkin’s achievement, either, for this is indeed the whole point of his poetry: to write, not about the slums, the canal or the church, but about the elegance of melancholy.

Why on earth should melancholy be elegant – or attractive in any other way? On the face of it, it ought to be precisely the sort of thing that evolution breeds out of the race, a prime target for sexual deselection. What female would want to mate with a miserable partner when she could have a happy, smiling one instead? Put like that, of course, the question looks a little ridiculous; as if we’d really prefer to pair off with SpongeBob SquarePants instead of Morrissey. But why? Why would you rather spend time with the latter than the former?

If depression is a foul miasma wreathing the brain, elegant sadness is more like a peacock’s tail, coloured in blue-gentian and rich marine greens

It was Charles Darwin, in The Expression of the Emotions in Man and Animals (1872), who noted that sadness manifested the same way in all cultures. For something so ubiquitous, it is tempting to venture an evolutionary explanation. Alas, the anthropological and evolutionary work in this area has focused almost entirely upon depression, which is not quite what we are talking about here. I can tell you with rather grim authority that the difference between elegant ennui and the black dog is like the difference between pleasant intoxication and typhus. Many evolutionary theories have been proposed for depression’s adaptive value, but no one has, so far as I am aware, tried to claim that it is enjoyable.

If depression is a foul miasma wreathing the brain, elegant sadness is more like a peacock’s tail, coloured in blue-gentian and rich marine greens. Is it also universal? To this question, anthropology offers no definitive answer. Yet the condition certainly manifests itself in a suggestive array of cultures. It is the sadness to which the Japanese phrase mono no aware gestures (物の哀れ, literally ‘the beautiful sorrow of things’). It is the haunted simplicity of those musical traditions that spread from Africa into the New World as the Blues. It’s the mixture of strength, energy, pity and melancholy that Claude Lévi-Strauss found in Brazil, encapsulated in the title of his book about his travels there Tristes Tropiques (1955). It’s the insight of Vergil’s Aeneas, as he looks back over his troubled life and forward to troubles yet to some: sunt lacrimae rerum; there are tears in everything, said not mournfully nor hopelessly but as a paradoxical statement about the beauty of the world (Aeneid 1:462).

It would be possible, of course, to construct a ‘cost benefit analysis’ of the sorts of sadness I am describing here. We might suggest that it is a signal that the individual in question has the strength, leisure and sensitivity to indulge in being sad. Saying so invokes what evolutionary scientists call ‘the handicap principle’, a hypothesis first framed by the Israeli evolutionary biologist Amotz Zahavi in 1975. The idea is that extravagant traits such as the highland deer’s massive antlers or the peacock’s tail are useful because they are so ostentatiously expensive, manifestly inconveniencing the owner. They are a way of saying: I’m so strong, my genes are so desirable, that I can afford to schlep about with this manifest – and, by the way, beautiful – disadvantage attached to my body.

Sadness, according to this model, is a kind of conspicuous consumption. It takes more muscles to frown than smile, and maybe that’s the point. It signals ones capacity to squander a resource precisely by squandering it. Any fool can live and be happy. It takes greater strength to live and be sad.

All the same, this analysis loses the most important aspect of this emotion; not that it costs, but that it is beautiful. Happy can be pretty, but some species of sad have access to beauties that happy can never know.