Should scholars serve the truth, even at the cost of social harmony? Should you expose a fiction even if that fiction sustains the social order? In writing my latest book, 21 Lessons for the 21st Century, I had to struggle with this dilemma with regard to liberalism.

On the one hand, I believe that the liberal story is flawed, that it does not tell the truth about humanity, and that in order to survive and flourish in the 21st century we need to go beyond it. On the other hand, at present the liberal story is still fundamental to the functioning of the global order. What’s more, liberalism is now attacked by religious and nationalist fanatics who believe in nostalgic fantasies that are far more dangerous and harmful.

So should I speak my mind openly, risking that my words could be taken out of context and used by demagogues and autocrats to further attack the liberal order? Or should I censor myself? It is a mark of illiberal regimes that they make free speech more difficult even outside their borders. Due to the spread of such regimes, it is becoming increasingly dangerous to think critically about the future of our species.

I eventually chose free discussion over self-censorship, thanks to my belief both in the strength of liberal democracy and in the necessity to revamp it. Liberalism’s great advantage over other ideologies is that it is flexible and undogmatic. It can sustain criticism better than any other social order. Indeed, it is the only social order that allows people to question even its own foundations. Liberalism has already survived three big crises – the first world war, the fascist challenge in the 1930s, and the communist challenge in the 1950s-70s. If you think liberalism is in trouble now, just remember how much worse things were in 1918, 1938 or 1968.

The main challenge liberalism faces today comes not from fascism or communism but from the laboratories

In 1968, liberal democracies seemed to be an endangered species, and even within their own borders they were rocked by riots, assassinations, terrorist attacks and fierce ideological battles. If you happened to be amid the riots in Washington on the day after Martin Luther King was assassinated, or in Paris in May 1968, or at the Democratic party’s convention in Chicago in August 1968, you might well have thought that the end was near. While Washington, Paris and Chicago were descending into chaos, Moscow and Leningrad were tranquil, and the Soviet system seemed destined to endure for ever. Yet 20 years later it was the Soviet system that collapsed. The clashes of the 1960s strengthened liberal democracy, while the stifling climate in the Soviet bloc presaged its demise.

So we hope liberalism can reinvent itself yet again. But the main challenge it faces today comes not from fascism or communism, and not even from the demagogues and autocrats that are spreading everywhere like frogs after the rains. This time the main challenge emerges from the laboratories.

Liberalism is founded on the belief in human liberty. Unlike rats and monkeys, human beings are supposed to have “free will”. This is what makes human feelings and human choices the ultimate moral and political authority in the world. Liberalism tells us that the voter knows best, that the customer is always right, and that we should think for ourselves and follow our hearts.

Unfortunately, “free will” isn’t a scientific reality. It is a myth inherited from Christian theology. Theologians developed the idea of “free will” to explain why God is right to punish sinners for their bad choices and reward saints for their good choices. If our choices aren’t made freely, why should God punish or reward us for them? According to the theologians, it is reasonable for God to do so, because our choices reflect the free will of our eternal souls, which are independent of all physical and biological constraints.

This myth has little to do with what science now teaches us about Homo sapiens and other animals. Humans certainly have a will – but it isn’t free. You cannot decide what desires you have. You don’t decide to be introvert or extrovert, easy-going or anxious, gay or straight. Humans make choices – but they are never independent choices. Every choice depends on a lot of biological, social and personal conditions that you cannot determine for yourself. I can choose what to eat, whom to marry and whom to vote for, but these choices are determined in part by my genes, my biochemistry, my gender, my family background, my national culture, etc – and I didn’t choose which genes or family to have.

Hacked … biometric sensors could allow corporations direct access to your inner world. Photograph: Alamy Stock Photo

This is not abstract theory. You can witness this easily. Just observe the next thought that pops up in your mind. Where did it come from? Did you freely choose to think it? Obviously not. If you carefully observe your own mind, you come to realise that you have little control of what’s going on there, and you are not choosing freely what to think, what to feel, and what to want.

Though “free will” was always a myth, in previous centuries it was a helpful one. It emboldened people who had to fight against the Inquisition, the divine right of kings, the KGB and the KKK. The myth also carried few costs. In 1776 or 1945 there was relatively little harm in believing that your feelings and choices were the product of some “free will” rather than the result of biochemistry and neurology.

But now the belief in “free will” suddenly becomes dangerous. If governments and corporations succeed in hacking the human animal, the easiest people to manipulate will be those who believe in free will.

In order to successfully hack humans, you need two things: a good understanding of biology, and a lot of computing power. The Inquisition and the KGB lacked this knowledge and power. But soon, corporations and governments might have both, and once they can hack you, they can not only predict your choices, but also reengineer your feelings. To do so, corporations and governments will not need to know you perfectly. That is impossible. They will just have to know you a little better than you know yourself. And that is not impossible, because most people don’t know themselves very well.

If you believe in the traditional liberal story, you will be tempted simply to dismiss this challenge. “No, it will never happen. Nobody will ever manage to hack the human spirit, because there is something there that goes far beyond genes, neurons and algorithms. Nobody could successfully predict and manipulate my choices, because my choices reflect my free will.” Unfortunately, dismissing the challenge won’t make it go away. It will just make you more vulnerable to it.

It starts with simple things. As you surf the internet, a headline catches your eye: “Immigrant gang rapes local women”. You click on it. At exactly the same moment, your neighbour is surfing the internet too, and a different headline catches her eye: “Trump prepares nuclear strike on Iran”. She clicks on it. Both headlines are fake news stories, generated perhaps by Russian trolls, or by a website keen on increasing traffic to boost its ad revenues. Both you and your neighbour feel that you clicked on these headlines out of your free will. But in fact you have been hacked.

If governments succeed in hacking the human animal, the easiest people to manipulate will be those who believe in free will

Propaganda and manipulation are nothing new, of course. But whereas in the past they worked like carpet bombing, now they are becoming precision-guided munitions. When Hitler gave a speech on the radio, he aimed at the lowest common denominator, because he couldn’t tailor his message to the unique weaknesses of individual brains. Now it has become possible to do exactly that. An algorithm can tell that you already have a bias against immigrants, while your neighbour already dislikes Trump, which is why you see one headline while your neighbour sees an altogether different one. In recent years some of the smartest people in the world have worked on hacking the human brain in order to make you click on ads and sell you stuff. Now these methods are being used to sell you politicians and ideologies, too.

And this is just the beginning. At present, the hackers rely on analysing signals and actions in the outside world: the products you buy, the places you visit, the words you search for online. Yet within a few years biometric sensors could give hackers direct access to your inner world, and they could observe what’s going on inside your heart. Not the metaphorical heart beloved by liberal fantasies, but rather the muscular pump that regulates your blood pressure and much of your brain activity. The hackers could then correlate your heart rate with your credit card data, and your blood pressure with your search history. What would the Inquisition and the KGB have done with biometric bracelets that constantly monitor your moods and affections? Stay tuned.

Liberalism has developed an impressive arsenal of arguments and institutions to defend individual freedoms against external attacks from oppressive governments and bigoted religions, but it is unprepared for a situation when individual freedom is subverted from within, and when the very concepts of “individual” and “freedom” no longer make much sense. In order to survive and prosper in the 21st century, we need to leave behind the naive view of humans as free individuals – a view inherited from Christian theology as much as from the modern Enlightenment – and come to terms with what humans really are: hackable animals. We need to know ourselves better.

Of course, this is hardly new advice. From ancient times, sages and saints repeatedly advised people to “know thyself”. Yet in the days of Socrates, the Buddha and Confucius, you didn’t have real competition. If you neglected to know yourself, you were still a black box to the rest of humanity. In contrast, you now have competition. As you read these lines, governments and corporations are striving to hack you. If they get to know you better than you know yourself, they can then sell you anything they want – be it a product or a politician.

It is particularly important to get to know your weaknesses. They are the main tools of those who try to hack you. Computers are hacked through pre-existing faulty code lines. Humans are hacked through pre-existing fears, hatreds, biases and cravings. Hackers cannot create fear or hatred out of nothing. But when they discover what people already fear and hate it is easy to push the relevant emotional buttons and provoke even greater fury.

If people cannot get to know themselves by their own efforts, perhaps the same technology the hackers use can be turned around and serve to protect us. Just as your computer has an antivirus program that screens for malware, maybe we need an antivirus for the brain. Your AI sidekick will learn by experience that you have a particular weakness – whether for funny cat videos or for infuriating Trump stories – and would block them on your behalf.

You feel that you clicked on these headlines out of your free will, but in fact you have been hacked. Photograph: Getty images

But all this is really just a side issue. If humans are hackable animals, and if our choices and opinions don’t reflect our free will, what should the point of politics be? For 300 years, liberal ideals inspired a political project that aimed to give as many individuals as possible the ability to pursue their dreams and fulfil their desires. We are now closer than ever to realising this aim – but we are also closer than ever to realising that this has all been based on an illusion. The very same technologies that we have invented to help individuals pursue their dreams also make it possible to re-engineer those dreams. So how can I trust any of my dreams?

From one perspective, this discovery gives humans an entirely new kind of freedom. Previously, we identified very strongly with our desires, and sought the freedom to realise them. Whenever any thought appeared in the mind, we rushed to do its bidding. We spent our days running around like crazy, carried by a furious rollercoaster of thoughts, feelings and desires, which we mistakenly believed represented our free will. What happens if we stop identifying with this rollercoaster? What happens when we carefully observe the next thought that pops up in our mind and ask: “Where did that come from?”

For starters, realising that our thoughts and desires don’t reflect our free will can help us become less obsessive about them. If I see myself as an entirely free agent, choosing my desires in complete independence from the world, it creates a barrier between me and all other entities. I don’t really need any of those other entities – I am independent. It simultaneously bestows enormous importance on my every whim – after all, I chose this particular desire out of all possible desires in the universe. Once we give so much importance to our desires, we naturally try to control and shape the whole world according to them. We wage wars, cut down forests and unbalance the entire ecosystem in pursuit of our whims. But if we understood that our desires are not the outcome of free choice, we would hopefully be less preoccupied with them, and would also feel more connected to the rest of the world.

If we understood that our desires are not the outcome of free choice, we would hopefully be less preoccupied with them

People sometimes imagine that if we renounce our belief in “free will”, we will become completely apathetic, and just curl up in some corner and starve to death. In fact, renouncing this illusion can have two opposite effects: first, it can create a far stronger link with the rest of the world, and make you more attentive to your environment and to the needs and wishes of others. It is like when you have a conversation with someone. If you focus on what you want to say, you hardly really listen. You just wait for the opportunity to give the other person a piece of your mind. But when you put your own thoughts aside, you can suddenly hear other people.

Second, renouncing the myth of free will can kindle a profound curiosity. If you strongly identify with the thoughts and desires that emerge in your mind, you don’t need to make much effort to get to know yourself. You think you already know exactly who you are. But once you realise “Hi, this isn’t me. This is just some changing biochemical phenomenon!” then you also realise you have no idea who – or what – you actually are. This can be the beginning of the most exciting journey of discovery any human can undertake.

There is nothing new about doubting free will or about exploring the true nature of humanity. We humans have had this discussion a thousand times before. But we never had the technology before. And the technology changes everything. Ancient problems of philosophy are now becoming practical problems of engineering and politics. And while philosophers are very patient people – they can argue about something inconclusively for 3,000 years – engineers are far less patient. Politicians are the least patient of all.

How does liberal democracy function in an era when governments and corporations can hack humans? What’s left of the beliefs that “the voter knows best” and “the customer is always right”? How do you live when you realise that you are a hackable animal, that your heart might be a government agent, that your amygdala might be working for Putin, and that the next thought that emerges in your mind might well be the result of some algorithm that knows you better than you know yourself? These are the most interesting questions humanity now faces.

Unfortunately, these are not the questions most humans ask. Instead of exploring what awaits us beyond the illusion of “free will”, people all over the world are now retreating to find shelter with even older illusions. Instead of confronting the challenge of AI and bioengineering, many are turning to religious and nationalist fantasies that are even less in touch with the scientific realities of our time than liberalism. Instead of fresh political models, what’s on offer are repackaged leftovers from the 20th century or even the middle ages.

When you try to engage with these nostalgic fantasies, you find yourself debating such things as the veracity of the Bible and the sanctity of the nation (especially if you happen, like me, to live in a place like Israel). As a scholar, this is a disappointment. Arguing about the Bible was hot stuff in the age of Voltaire, and debating the merits of nationalism was cutting-edge philosophy a century ago – but in 2018 it seems a terrible waste of time. AI and bioengineering are about to change the course of evolution itself, and we have just a few decades to figure out what to do with them. I don’t know where the answers will come from, but they are definitely not coming from a collection of stories written thousands of years ago.

So what to do? We need to fight on two fronts simultaneously. We should defend liberal democracy, not only because it has proved to be a more benign form of government than any of its alternatives, but also because it places the fewest limitations on debating the future of humanity. At the same time, we need to question the traditional assumptions of liberalism, and develop a new political project that is better in line with the scientific realities and technological powers of the 21st century.

Greek mythology tells that Zeus and Poseidon, two of the greatest gods, competed for the hand of the goddess Thetis. But when they heard the prophecy that Thetis would bear a son more powerful than his father, both withdrew in alarm. Since gods plan on sticking around for ever, they don’t want a more powerful offspring to compete with them. So Thetis married a mortal, King Peleus, and gave birth to Achilles. Mortals do like their children to outshine them. This myth might teach us something important. Autocrats who plan to rule in perpetuity don’t like to encourage the birth of ideas that might displace them. But liberal democracies inspire the creation of new visions, even at the price of questioning their own foundations •

• Yuval Noah Harari’s 21 Lessons for the 21st Century is published by Cape. To order a copy for £13.99, saving £5, go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.