We are far more powerful than our ancestors, but are we much happier? Historians seldom stop to ponder this question, yet ultimately, isn't it what history is all about? Our understanding and our judgment of, say, the worldwide spread of monotheistic religion surely depends on whether we conclude that it raised or lowered global happiness levels. And if the spread of monotheism had no noticeable impact on global happiness, what difference did it make?

With the rise of individualism and the decline of collectivist ideologies, happiness is arguably becoming our supreme value. With the stupendous growth in human production, happiness is also acquiring unprecedented economic importance. Consumerist economies are increasingly geared to supply happiness rather than subsistence or even affluence, and a chorus of voices is now calling for a replacement of GDP measurements with happiness statistics as the basic economic yardstick. Politics seems to be following suit. The traditional right to "the pursuit of happiness" is imperceptibly morphing into a right to happiness, which means that it is becoming the duty of government to ensure the happiness of its citizens. In 2007 the European commission launched "Beyond GDP" to consider the feasibility of using a wellbeing index to replace or complement GDP. Similar initiatives have recently been developed in numerous other countries – from Thailand to Canada, from Israel to Brazil.

Most governments still focus on achieving economic growth, but when asked what is so good about growth, even diehard capitalists almost invariably turn to happiness. Suppose we caught David Cameron in a corner, and demanded to know why he cared so much about economic growth. "Well," he might answer, "growth is essential to provide people with higher standards of living, better medical care, bigger houses, faster cars, tastier ice-cream." And, we could press further, what is so good about higher standards of living? "Isn't it obvious?" Cameron might reply, "It makes people happier."

A reason to be happy … tastier ice-cream? Photograph: Envision/Corbis

Suppose, for the sake of argument, that we could somehow scientifically prove that higher standards of living did not translate into greater happiness. "But David," we could say, "look at these historical, psychological and biological studies. They prove beyond any reasonable doubt that having bigger houses, tastier ice-cream and even better medicines does not increase human happiness." "Really?" he would gasp, "Why did nobody tell me! Well, if that's the case, forget about my plans to boost economic growth. I am leaving everything and joining a hippie commune."

This is a highly unlikely scenario, and not only because so far we have almost no scientific studies of the long-term history of happiness. Scholars have researched the history of just about everything – politics, economics, diseases, sexuality, food – yet they have seldom asked how they all influence human happiness. Over the last decade, I have been writing a history of humankind, tracking down the transformation of our species from an insignificant African ape into the master of the planet. It was not easy to understand what turned Homo sapiens into an ecological serial killer; why men dominated women in most human societies; or why capitalism became the most successful religion ever. It wasn't easy to address such questions because scholars have offered so many different and conflicting answers. In contrast, when it came to assessing the bottom line – whether thousands of years of inventions and discoveries have made us happier – it was surprising to realise that scholars have neglected even to ask the question. This is the largest lacuna in our understanding of history.

The Whig view of history

Though few scholars have studied the long-term history of happiness, almost everybody has some idea about it. One common preconception – often termed "the Whig view of history" – sees history as the triumphal march of progress. Each passing millennium witnessed new discoveries: agriculture, the wheel, writing, print, steam engines, antibiotics. Humans generally use newly found powers to alleviate miseries and fulfil aspirations. It follows that the exponential growth in human power must have resulted in an exponential growth in happiness. Modern people are happier than medieval people, and medieval people were happier than stone age people.

The discovery of antibiotics … a marker in the triumphal march of progess. Photograph: Doug Steley C/Alamy

But this progressive view is highly controversial. Though few would dispute the fact that human power has been growing since the dawn of history, it is far less clear that power correlates with happiness. The advent of agriculture, for example, increased the collective power of humankind by several orders of magnitude. Yet it did not necessarily improve the lot of the individual. For millions of years, human bodies and minds were adapted to running after gazelles, climbing trees to pick apples, and sniffing here and there in search of mushrooms. Peasant life, in contrast, included long hours of agricultural drudgery: ploughing, weeding, harvesting and carrying water buckets from the river. Such a lifestyle was harmful to human backs, knees and joints, and numbing to the human mind.

In return for all this hard work, peasants usually had a worse diet than hunter-gatherers, and suffered more from malnutrition and starvation. Their crowded settlements became hotbeds for new infectious diseases, most of which originated in domesticated farm animals. Agriculture also opened the way for social stratification, exploitation and possibly patriarchy. From the viewpoint of individual happiness, the "agricultural revolution" was, in the words of the scientist Jared Diamond, "the worst mistake in the history of the human race".

The case of the agricultural revolution is not a single aberration, however. Themarch of progress from the first Sumerian city-states to the empires of Assyria and Babylonia was accompanied by a steady deterioration in the social status and economic freedom of women. The European Renaissance, for all its marvellous discoveries and inventions, benefited few people outside the circle of male elites. The spread of European empires fostered the exchange of technologies, ideas and products, yet this was hardly good news for millions of Native Americans, Africans and Aboriginal Australians.

The point need not be elaborated further. Scholars have thrashed the Whig view of history so thoroughly, that the only question left is: why do so many people still believe in it?

Paradise lost

There is an equally common but completely opposite preconception, which might be dubbed the "romantic view of history". This argues that there is a reverse correlation between power and happiness. As humankind gained more power, it created a cold mechanistic world, which is ill-suited to our real needs.

'Computers have turned us into zombies.' Photograph: Christopher Thomond for the Guardian

Romantics never tire of finding the dark side of every discovery. Writing gave rise to extortionate taxation. Printing begot mass propaganda and brainwashing. Computers turn us into zombies. The harshest criticism of all is reserved for the unholy trinity of industrialism, capitalism and consumerism. These three bugbears have alienated people from their natural surroundings, from their human communities, and even from their daily activities. The factory worker is nothing but a mechanical cog, a slave to the requirements of machines and the interests of money. The middle class may enjoy better working conditions and many material comforts, but it pays for them dearly with social disintegration and spiritual emptiness. From a romantic perspective, the lives of medieval peasants were preferable to those of modern factory-hands and office clerks, and the lives of stone-age foragers were the best of all.

Yet the romantic insistence on seeing the dark side of every novelty is as dogmatic as the Whig belief in progress. For instance, over the last two centuries modern medicine has beaten back the army of diseases that prey on humankind, from tuberculosis and measles to cholera and diphtheria. Average life expectancy has soared, and global child mortality has dropped from roughly 33% to less than 5%. Can anyone doubt that this made a huge contribution to the happiness not only of those children who might otherwise be dead, but also of their parents, siblings and friends?

Paradise now

A more nuanced stance agrees with the romantics that, up until the modern age, there was no clear correlation between power and happiness. Medieval peasants may indeed have been more miserable than their hunter-gatherer ancestors. But the romantics are wrong in their harsh judgment of modernity. In the last few centuries we have not only gained immense powers, but more importantly, new humanist ideologies have finally harnessed our collective power in the service of individual happiness. Despite some catastrophes such as the Holocaust and the Atlantic slave trade (so the story goes), we have at long last turned the corner and begun increasing global happiness systematically. The triumphs of modern medicine are just one example. Other unprecedented achievements include the decline of international wars; the dramatic drop in domestic violence; and the elimination of mass-scale famines. (See Steven Pinker's book The Better Angels of Our Nature.)

Yet this, too, is an oversimplification. We can congratulate ourselves on the accomplishments of modern Homo sapiens only if we completely ignore the fate of all other animals. Much of the wealth that shields humans from disease and famine was accumulated at the expense of laboratory monkeys, dairy cows and conveyor-belt chickens. Tens of billions of them have been subjected over the last two centuries to a regime of industrial exploitation, whose cruelty has no precedent in the annals of planet Earth.

'Much of the wealth that shields humans from disease and famine was accumulated at the expense of laboratory monkeys, dairy cows and conveyor-belt chickens.' Photograph: Graeme Robertson for the Guardian

Secondly, the time frame we are talking about is extremely short. Even if we focus only on the fate of humans, it is hard to argue that the life of the ordinary Welsh coalminer or Chinese peasant in 1800 was better than that of the ordinary forager 20,000 years ago. Most humans began to enjoy the fruits of modern medicine no earlier than 1850. Mass famines and major wars continued to blight much of humanity up to the middle of the 20th century. Even though the last few decades have proven to be a relative golden age for humanity in the developed world, it is too early to know whether this represents a fundamental shift in the currents of history, or an ephemeral wave of good fortune: 50 years is simply not enough time on which to base sweeping generalisations.

Indeed, the contemporary golden age may turn out to have sown the seeds of future catastrophe. Over the last few decades we have been disturbing the ecologic equilibrium of our planet in myriad ways, and nobody knows what the consequences will be. We may be destroying the groundwork of human prosperity in an orgy of reckless consumption.

Lonely and grey?

Even if we take into account solely the citizens of today's affluent societies, Romantics may point out that our comfort and security have their price. Homo sapiens evolved as a social animal, and our wellbeing is usually influenced by the quality of our relationships more than by our household amenities, the size of our bank accounts or even our health. Unfortunately, the immense improvement in material conditions that affluent westerners have enjoyed over the last century was coupled with the collapse of most intimate communities.

People in the developed world rely on the state and the market for almost everything they need: food, shelter, education, health, security. Therefore it has become possible to survive without having extended families or any real friends. A person living in a London high rise is surrounded by thousands of people wherever she goes, but she might never have visited the flat next door, and might know very little about her colleagues at work. Even her friends might be just pub buddies. Many present-day friendships involve little more than talking and having fun together. We meet a friend at a pub, call him on the phone, or send an email, so that we can unload our anger at what happened today in the office, or share our thoughts on the latest royal scandal. Yet how well can you really know a person only from conversations?

In contrast to such pub buddies, friends in the stone age depended on one another for their very survival. Humans lived in close-knit communities, and friends were people with whom you went hunting mammoths. You survived long journeys and difficult winters together. You took care of one another when one of you fell sick, and shared your last morsels of food in times of want. Such friends knew each other more intimately than many present-day couples. Replacing such precarious tribal networks with the security of modern economies and states obviously has enormous advantages. But the quality and depth of intimate relationships are likely to have suffered.

In the stone age, friends went mammoth-hunting together. Photograph: Andrew Nelmerm/Getty Images/Dorling Kindersley

In addition to shallower relationships, contemporary people also suffer from a much poorer sensory world. Ancient foragers lived in the present moment, acutely aware of every sound, taste and smell. Their survival depended on it. They listened to the slightest movement in the grass to learn whether a snake might be lurking there. They carefully observed the foliage of trees in order to discover fruits and birds' nests. They sniffed the wind for approaching danger. They moved with a minimum of effort and noise, and knew how to sit, walk and run in the most agile and efficient manner. Varied and constant use of their bodies gave them physical dexterity that people today are unable to achieve even after years of practising yoga or tai chi.

Today we can go to the supermarket and choose to eat a thousand different dishes. But whatever we choose, we might eat it in haste in front of the TV, not really paying attention to the taste. We can go on vacation to a thousand amazing locations. But wherever we go, we might play with our smartphone instead of really seeing the place. We have more choice than ever before, but what good is this choice, when we have lost the ability really to pay attention?

Well, what did you expect?

Even if you don't buy into this picture of Pleistocene richness replaced by modern poverty, it is clear that the immense rise in human power has not been matched by an equal rise in human happiness. We are a thousand times more powerful than our hunter-gatherer ancestors, but not even the most optimistic Whig can believe that we are a thousand times happier. If we told our great-great-grandmother how we live, with vaccinations and painkillers and running water and stuffed refrigerators, she would likely have clasped her hands in astonishment and said: "You are living in paradise! You probably wake up every morning with a song in your heart, and pass your days walking on sunshine, full of gratitude and loving-kindness for all." Well, we don't. Compared to what most people in history dreamed about, we may be living in paradise. But for some reason, we don't feel that we are.

One explanation has been provided by social scientists, who have recently rediscovered an ancient wisdom: our happiness depends less on objective conditions and more on our own expectations. Expectations, however, tend to adapt to conditions. When things improve, expectations rise, and consequently even dramatic improvements in conditions might leave us as dissatisfied as before. In their pursuit of happiness, people are stuck on the proverbial "hedonic treadmill", running faster and faster but getting nowhere.

If you don't believe that, just ask Hosni Mubarak. The average Egyptian was far less likely to die from starvation, plague or violence under Mubarak than under any previous regime in Egyptian history. In all likelihood, Mubarak's regime was also less corrupt. Nevertheless, in 2011 Egyptians took to the streets in anger to overthrow Mubarak. For they had much higher expectations than their ancestors.

Indeed, if happiness is strongly influenced by expectations then one of the central pillars of the modern world, mass media, seems almost tailored to prevent significant increases in global happiness levels. A man living in a small village 5,000 years ago measured himself against the other 50 men in the village. Compared to them, he looked pretty hot. Today, a man living in a small village compares himself to film stars and models, whom he sees every day on screens and giant billboards. Our modern villager is likely to be less happy with the way he looks.

The biological glass ceiling

Evolutionary biologists offer a complementary explanation for the hedonic treadmill. They contend that both our expectations and our happiness are not really determined by political, social or cultural factors, but rather by our biochemical system. Nobody is ever made happy, they argue, by getting a promotion, winning the lottery, or even finding true love. People are made happy by one thing and one thing only – pleasant sensations in their bodies. A person who just got a promotion and jumps from joy is not really reacting to the good news. She is reacting to various hormones coursing through her bloodstream, and to the storm of electric signals flashing between different parts of her brain.

'Evolution has no interest in happiness per se: it is interested only in survival and repro­duction, and it uses happiness and misery as mere goads.' Photograph: Philipp Kammerer/Alamy

The bad news is that pleasant sensations quickly subside. If last year I got a promotion, I might still be filling the new position, but the very pleasant sensations that I felt back then subsided long ago. If I want to continue feeling such sensations, I must get another promotion. And another. This is all the fault of evolution. Evolution has no interest in happiness per se: it is interested only in survival and reproduction, and it uses happiness and misery as mere goads. Evolution makes sure that no matter what we achieve, we remain dissatisfied, forever grasping for more. Happiness is thus a homeostatic system. Just as our biochemical system maintains our body temperature and sugar levels within narrow boundaries, it also prevents our happiness levels from rising beyond certain thresholds.

If happiness really is determined by our biochemical system, then further economic growth, social reforms and political revolutions will not make our world a much happier place. The only way dramatically to raise global happiness levels is by psychiatric drugs, genetic engineering and other direct manipulations of our biochemical infrastructure. In Brave New World, Aldous Huxley envisaged a world in which happiness is the supreme value, and in which everybody constantly takes the drug soma, which makes people happy without harming their productivity and efficiency. The drug forms one of the foundations of the World State, which is never threatened by wars, revolutions or strikes, because all people are supremely content with their current conditions. Huxley presented this world as a frightening dystopia. Today, more and more scientists, policymakers and ordinary people are adopting it as their goal.

Second thoughts

There are people who think that happiness is really not that important, and that it is a mistake to define individual satisfaction as the aim of human society. Others agree that happiness is the supreme good, but think that happiness isn't just a matter of pleasant sensations. Thousands of years ago Buddhist monks reached the surprising conclusion that pursuing pleasant sensations is in fact the root of suffering, and that happiness lies in the opposite direction. Pleasant sensations are just ephemeral and meaningless vibrations. If five minutes ago I felt joyful or peaceful, that feeling is now gone, and I may well feel angry or bored. If I identify happiness with pleasant sensations, and crave to experience more and more of them, I have no choice but constantly to pursue them, and even if I get them, they immediately disappear, and I have to start all over again. This pursuit brings no lasting achievement. On the contrary: the more I crave these pleasant sensations, the more stressed and dissatisfied I become. However, if I learn to see my sensations for what they really are – ephemeral and meaningless vibrations – I lose interest in pursuing them, and can be content with whatever I experience. For what is the point of running after something that disappears as fast as it arises? For Buddhism, then, happiness isn't pleasant sensations, but rather the wisdom, serenity and freedom that come from understanding our true nature.

True or false, the practical impact of such alternative views is minimal. For the capitalist juggernaut, happiness is pleasure. Full stop. With each passing year, our tolerance for unpleasant sensations decreases, whereas our craving for pleasant sensations increases. Both scientific research and economic activity are geared to that end, producing each year better painkillers, new ice-cream flavours, more comfortable mattresses, and more addictive games for our smartphones, so that we will not suffer a single boring moment while waiting for the bus.

All this is hardly enough, of course. Humans are not adapted by evolution to experience constant pleasure, so ice‑cream and smartphone games will not do. If that is what humankind nevertheless wants, it will be necessary to re-engineer our bodies and minds. We are working on it.

• Yuval Noah Harari's Sapiens: A Brief History of Humankind is published by Harvill Secker. He will be speaking at the Natural History Museum, Oxford, on 10 September.