There has, over the past couple of decades, been a remarkable boom in economic research into happiness. Strangely enough, it might have originated in remarks made in the early 1970s by Jigme Singye Wangchuck at the time of his coronation as absolute monarch of Bhutan, one of the poorest countries on Earth. Questioned about policies to promote growth in Gross National Product, the King said he would rather strive to promote Gross National Happiness than the conventional goal of Gross National Product. This neat aphorism spurred on a full-scale research agenda, not only in Bhutan but in the developed world as well. Over time, a new group of academic stars – if not household names – has emerged, not least economists such as Richard Easterlin, Bruno Frey and Richard Layard.

And what have we learnt so far? Chiefly this: that there is a reason why everyone reads Dante’s ‘Inferno’ and no one reads his ‘Paradiso’. ‘All happy families are alike,’ Tolstoy wrote; he might have added that this is why so few of them appear in novels. Maybe there just isn’t anything interesting to say about happiness. That certainly seems true of the social science work on it, where even the most basic results look like shadows cast by the analytical framework rather than genuine discoveries.

Take the field’s crucial finding, the so-called Easterlin paradox. Cross-country data seem to show pretty consistently that, on average, happiness increases with income, but only up to a certain point.* In the developed world, for example, people are scarcely happier than they were in the 1960s. The evidence for this claim consists of surveys in which people rate their happiness on a scale, typically from one to 10. Within any given society, happiness tends to rise with all the obvious variables: income, health, family relationships and so on. But between societies, or in Western societies such as Australia over time, there’s not much movement, even though both income and health have improved pretty steadily for a long time.

This sounds like quite a discovery: happiness is relative! But in fact the result is probably an illusion. To see why, just consider this puzzle. Suppose you wanted to establish whether children’s height increased with age, but for some reason you couldn’t measure them directly. One way to start the investigation would be to interview groups of children in different classes at school and ask them the question: ‘On a scale of one to 10, how tall are you?’ You’d find that kids who were old relative to their classmates would tend to report higher numbers than those who were young relative to their classmates. After all, the older ones would mostly be taller than the younger ones. So far, so good.

After you had surveyed a few classes, you might start to notice something rather odd. As you moved up through the year groups, the average age would keep increasing but the average reported height would not change much. For all age groups, the median response would be something like seven (accounting for the ‘Lake Wobegon’ effect, by which nearly everyone thinks of themselves as above average). What to make of that?

By analogy with the ‘happiness puzzle’, you might conclude that height is a subjective construct depending on relative, rather than absolute, age. But, in reality, we know that height actually does increase with age throughout childhood. The problem is that asking for a subjective mark out of 10 is a silly way to measure height. Each child is likely to score himself relative to his classmates rather than to any absolute scale, with the result that comparisons between age groups are meaningless. Does happiness keep rising with income? Nobody can say. But, since we don’t have any absolute scale of mood, it certainly seems plausible that people judge it in pretty much the same way as the children judged height in our imaginary investigation.

In a society where most people are hungry most of the time, having a full belly might justify a pretty decent happiness score. If everyone has enough to eat but it’s mostly rice or potatoes, you might consider yourself blessed to be eating roast chicken, and so on. The objective level of income and health needed to report yourself as more than averagely happy will depend on what you consider average. This is true whether or not people in rich societies are in fact happier, and whether or not the average person is happier now than the average person in 1960. A relative scale tells us nothing one way or the other. And as far as happiness goes, a relative scale seems to be all that we can hope for.

Economists are famous for concentrating on problems for which they have data, rather than ones that matter. As an old joke has it, they are like the drunk man searching for his keys under a lamppost because the light is better there. So: happiness is intrinsically shadowy. Indeed, according to the so-called ‘hedonic paradox’, even those who hope to achieve it in their own lives can approach it only indirectly. As John Stuart Mill observed in 1873: ‘Aiming thus at something else, they find happiness by the way […] Ask yourself whether you are happy, and you cease to be so’.

Misery, by contrast, is a marvellously rich source of data. Unhappy families are, as Tolstoy pointed out, much more varied than happy ones. And if happiness is elusive and subjective, there are plenty of objective sources of unhappiness: hunger, illness, the premature death of loved ones, family breakdown and so on. We can measure the ways these things change over time and compare that data to subjective emotional evidence. A whole new research programme suggests itself.

When we shift our attention from unhappiness, some important political distinctions also come to light. Broadly speaking, everyone is in favour of happiness. It is a classic ‘motherhood’ issue (more so, indeed, than motherhood). Utilitarians seek to maximise it, even if they no longer believe in the precision of Jeremy Bentham’s ‘felicific calculus’. The classical liberals who wrote the US Declaration of Independence listed the pursuit of it among the inalienable rights of (white) men. Conservatives, according to a large body of scientific research, claim to be happier than liberals (in both the US and the European sense of ‘liberal’). Even the Nazis sought ‘strength through joy’. Of course, different political viewpoints yield different claims about what is most likely to promote happiness, but pretty much everyone is in favour of it. Even a book such as Eric Wilson’s Against Happiness (2008) turns out to be more accurately described by its subtitle: ‘In Praise of Melancholy’.

Unhappiness, by contrast, is divisive. Utilitarians take the commonsense view that, just as happiness is good and to be promoted, unhappiness is bad and should be minimised. This idea, coupled with the belief that the state should act to relieve unhappiness, is central to the philosophy of social democrats (and, with some qualifications, social liberals). The political right, by contrast, has a more equivocal view of unhappiness. Just consider the central dividing line of modern politics: welfare.

As an institution, the welfare state is not traditionally associated with very much happiness. If asked to list the sources of joy in our lives, few of us would include the receipt of unemployment benefits or a stay in a public hospital. But what it does (or tries to do) is minimise the sources of unhappiness in a market economy. It addresses illness, loss of income through unemployment or inability to work, homelessness and so on. This is not to say that, even in the most comprehensive system – one in which the fear of unemployment has been banished and there are decent public services for all – everyone is happy. Money doesn’t buy happiness. Still, as the comedian Spike Milligan said, it does buy a better class of unhappiness.

For many Americans, a better class of unhappiness would do nicely

And as far as that goes, the welfare state has had a remarkable track record. You just have to compare outcomes in nations that have modern welfare systems with those in the US, where the New Deal produced only stunted versions of the standard institutions. American technology leads the world and the US constitution endorses the pursuit of happiness. Yet on numerous important development indices, including premature mortality, food security, incarceration and access to health care, the US lags behind much of the rest of the developed world. For many Americans, a better class of unhappiness would do nicely.

Despite its achievements, the welfare state has never won much love on the political right. Why is that? Ostensibly, the chief point in the right-wing critique is that it creates more misery than it relieves in the long run. This argument generally depends on claims about the adverse incentive effects of high taxes and welfare payments – claims that have not, in general, stood up very well to data. The literature on these topics is huge, but perhaps the most striking evidence is the plummeting proportion of adult Americans who are in work, despite the supposedly powerful incentives of low taxes and minimal welfare.

Underlying ostensibly practical objections however, we often find signs of a belief that unhappiness is actually good for us, or at least, good for those on whom it is imposed. This idea might draw support from the orthodox Christian vision of the present world as a vale of tears in which original sin must be expiated. More generally, conservatives frequently see unhappiness as both inevitable, given the futility of attempts to reform human nature, and a necessary part of building an upright character.

William Bennett, a representative figure, served as US education secretary during the Reagan administration. In The Book of Virtues: A Treasury of Great Moral Stories (1993), which he edited, Bennett finds the greatness of American character exemplified by (at least some of) the members of the Donner Party, a group of 19th-century pioneers who got trapped by snow in the Sierra Nevada. Many of them starved to death. Others survived only by eating the remains of their companions. As you might expect, the situation brought out both the best and the worst in people. But Bennett’s wistful admiration of the stoic survivors as paragons from a more virtuous age (‘Where did those people go?’ he wonders) shows just how deep the division is between conservative ‘virtue ethics’ and the consequentialist view that unhappiness is an evil to be minimised. Bertolt Brecht sums up this division neatly in his play Galileo (1943), meeting the virtue-ethicist line ‘Unhappy the land that has no heroes!’ with a consequentialist riposte: ‘No. Unhappy is the land that needs a hero.’

If traditional conservatives speak highly of unhappiness, an even rosier view can be found among Prometheans and radicals of both the right and left. For market liberals, for example, the idea that we could ever be satisfied with a given level of material comfort is anathema. Our unhappiness with what we have is exactly what makes us strive for more. The market economy that makes this possible therefore represents the highest possible stage of social evolution.

The American political writer Virginia Postrel is a particularly eloquent advocate for this system of insatiable desire. She has argued against broadly instrumentalist theories of conspicuous consumption that claim we seek luxuries in order to impress others and pursue social status. In Postrel’s view, consumers will seek luxury even in items that no one else will ever see, such as $400 toilet brushes. Her book The Power of Glamour (2013) presents glamour as a kind of ‘non-verbal rhetoric’ that ‘leads us to feel that the life we dream of exists, and to desire it even more’.

Unlike merely material needs, the need for glamour can never be assuaged. For the vast majority of us, the prize must remain forever out of reach. Nevertheless, Postrel argues, we are better off striving after what we know to be illusions than living with the dull contentment we find when our objective material needs are satisfied. (Those who starve while others dissipate resources in the pursuit of glamour might have a different view.)

Marxists argue that, by dulling the pain of poverty, welfare and religion diminish the anger that workers ought to feel at a system that exploits them

And it is only fair to note that not all of the champions of unhappiness are on the political right. Some on the Marxist left have argued that, like religion, social welfare is an opiate for the masses. By dulling the pain of poverty and unemployment, both welfare and religion diminish the anger that workers ought to feel at a system that exploits and alienates them. The revolutionary alternative would be to ‘heighten the contradictions’ of capitalism – by making it more miserable, until it is no longer tolerable.

For a brief moment in the 1960s, it appeared that this radical critique might succeed. A (largely hypothetical) alliance of downtrodden workers and postmaterialist student radicals would shatter the illusion of prosperous middle-class consensus and dispense with the palliatives of the welfare system, calling forth a new society in which genuine happiness for all might finally flourish.

In fact, this left-wing revolt achieved little except weakening the defences of the welfare state against the real challenge, which came from the free-market right. Nevertheless, a good deal of social criticism draws on the residual appeal of this critique, denouncing any ameliorative reform while implicitly conceding that there is nothing else on offer. One group that has taken this line to its logical conclusion is the former Revolutionary Communist Party of the UK, most of whose members are now associated with the market-libertarian group around Spiked magazine.

We are now at something of an impasse. Despite decades of pressure for cutbacks and rationalisation, the core institutions of the welfare state have endured. In most countries, what protects them is a sense on the part of the public that they are the only reliable bulwark against the miseries of unemployment, ill health and old age. Yet welfare’s opponents remain, and they are often, as we have seen, ideologically wedded to various forms of unhappiness, at least as experienced by others.

So, perhaps we need a new research programme, to examine how unhappiness really works. Does hunger, or unemployment, or the loss of a family member to preventable illness make you a stronger and better person? Is striving after more and better possessions more fulfilling than satisfaction with what you have? It’s obvious from the way I’ve posed these questions what I believe the answer to be. But genuine research into the economics of unhappiness might yield some surprising answers to such questions as these, and reveal new questions that we have never before considered.

Perhaps Bhutan will lead the way once again. Paternalist monarchy has given way to democratic elections, the second of which produced a surprise win for the People’s Democratic Party of Tshering Tobgay last year. Tellingly, he has abandoned the pursuit of Gross National Happiness. ‘Rather than talking about happiness,’ Tobgay told The New York Times, ‘we want to work on reducing the obstacles to happiness’. Such a pragmatic view might not make Tobgay a cultural icon like his predecessor, but it seems likely to do more to relieve the people of Bhutan from the various miseries attendant on poverty.

* Other researchers, such as Justin Wolfers and Betsey Stevenson, using different measures of happiness, have found a statistically significant, but relatively small, link between increased income and self-reported happiness over time.

Correction, April 1, 2014: When first published, this article placed the Donner Party in the Rocky Mountains. In fact they were stuck in the Sierra Nevada.