What will define the 21st century? When the question was put to a wide range of thinkers by Prospect magazine, the answers read like the horsemen of the apocalypse - disease, disaster, mayhem. Not cheerful bedtime reading then. The comments of philosopher, Jonathan Rée seemed to sum it all up: at the beginning of the 20th century, "the main emotion behind most people's politics was hope: hope for science, for free trade, for social democracy, for national efficiency, for world government". That sentiment has now been replaced, he argued, by indignation. "People are more interested in bearing witness to their personal moral righteousness" than in engaging in open-minded debate.

Optimism and a belief in progress are now the implausible preserve of Labour party apparatchiks who are regarded as at best deluded, at worst as cynically trying to preserve their own legitimacy. The rest of us have little faith in the capacity of human beings for self-sacrifice or cooperation to avert climate change or any of the other predicted catastrophes that fill the media.

Gloomy thoughts for a Monday morning. Last night the BBC television series The Trap: What Happened to Our Dream of Freedom began, claiming to explain how we have managed to land ourselves in this miasma of misery. Its director, Adam Curtis, has built a reputation on tracing how ideas shape political and social trends. This series, though his most dense, could be his most important yet. Ultimately, its message is optimistic - better understanding of the trap we're in will help us find a way out.

The central tenet of the argument is that during the cold war an understanding of human nature as suspicious, distrustful and always operating out of self-interest came to dominate political thinking. From that emerged a narrow definition of freedom as "giving people the ability to get whatever they wanted". This kind of freedom has become the central political idea of the past 25 years, but it's a corrosive form of pessimism rooted in a bleak, simplistic view of human nature.

It all goes back to the bizarre world of cold-war strategists in America developing sophisticated ways to achieve the "delicate balance of terror". They seized upon game theory that originated in poker playing as a way of rationally calculating your opponent's moves and therefore your own. How many Soviet cities would you have to nuke to deter the Soviets from nuking New York? The theory was that the suspicious distrustfulness of both sides in the cold war created a kind of stability.

If that was the case for nuclear weapons, perhaps the model could be applied elsewhere? John Nash, a mathematical genius at the US thinktank Rand and subject of the film A Beautiful Mind, took game theory further and developed the Nash equilibrium, which argued that the rational pursuit of self-interest by human beings could lead to a kind of social order. Selfishness didn't have to lead to social breakdown.

For the economist Friedrich von Hayek (Thatcher's inspiration) this was vindication of his belief that individual selfishness creates, spontaneously, "a self-directed automatic system". He told an interviewer, "altruism doesn't come into it"; just free up people's ability to pursue their self-interest and that will ultimately benefit everyone.

By the 70s, these ideas were being applied to politics by theorist James Buchanan, who argued that the notion of public duty was a sham used by bureaucracies and politicians to mask their own self-interest. There was no such thing as public good, he claimed, because that meant shared goals based on self-sacrifice, when what motivated people was their self-interest. The TV comedy series Yes Minister was based on Buchanan's public-choice theory, revealing a world of politics as pure calculation, spin and self-interest - which we now take for granted.

Initially, Buchanan's ideas offered politicians a new legitimacy. Three British prime ministers have used them to promise their electorates an illusion of more freedom. They have all offered to sweep away the self-interested elites who govern the country. Blair described Labour's goal in one conference speech: "To liberate people from old class divisions, old structures, old ways of working that will not do in this new world of change." This anti-elitism was seductive, the promise of individual freedom tempting - greater choice and greater autonomy have become the lodestar of politics of both left and right.

Just as public-choice theory was gaining ground in politics on both sides of the Atlantic, powerful reinforcement of its basic premise about the nature of human beings came from an unexpected quarter: genetics. Human beings were driven by genes, programmed for survival, Richard Dawkins's 1976 book The Selfish Gene argued. As Dawkins put it, "our DNA is an encoded description of the worlds in which our ancestors lived". We are blind creatures driven by genetic information millions of years old derived from short, brutish lives.

It doesn't get much more grim. In one telling clip in his series, Curtis asks Buchanan about where idealism comes in. "What do you mean by that?" Buchanan asks. "I can't get a handle on that."

This reading of human nature has ended up destroying the legitimacy of the political class that has espoused it, hollowing politics and all collective life out with the cynicism that we are like billiard balls bumping into each other without any common interest or capacity for collaboration. It's an ideology of freedom that has also created violent chaos. In Iraq, the US believed it had only to remove Saddam Hussein and liberate the people, and order would spontaneously emerge.

The original propagator of game theory, John Nash, has had second thoughts. In 1959 he developed paranoid schizophrenia and spent 10 years in mental hospitals. Now recovered, he admits he overemphasised the rationality of human beings and that not all behaviour is self-interested. Genetics is now moving into analysis of how cells select and edit DNA according to their environment; the idea that we are simply machines driven by DNA software is redundant.

There is always a time-lag between the world of ideas and politics, and we are still trapped in the cold-war mythology of human beings as rational and self-interested. We have lost faith in ourselves, in our humanity. But Curtis is optimistic. He believes the banality of the freedom that politicians have offered us for a generation is becoming clear - Iraq has painfully illustrated its absurdity to a global audience, while the freedom of the market has delivered growing inequality - and that will prompt a re-examination. But he acknowledges that one of the beneficiaries of our disillusionment with individual freedom will be a renaissance of conservative ideologies such as Islamism or Russian nationalism.

Such is the grip of cold-war mythology over our thinking that it's hard to share Curtis's optimism - the disillusionment is evident but not yet the new thinking that can overturn it. His diagnosis of our plight is riveting. Freedom - that cherished ideal so bankrupted by the frequent use of politicians and advertisers - needs to be reimagined.

· Part II of The Trap is broadcast on BBC2 next Sunday at 9pm

m.bunting@theguardian.com