‘Without the Cold War, what’s the point of being an American?” Harry “Rabbit” Angstrom, the novelist John Updike’s late-20th-century everyman, posed that question just as the “long twilight struggle” was winding down. More than quarter of a century later, the plaintive query still awaits a definitive answer.

Indeed, the passage of time has only sown confusion about whether there is a point to being an American. Even as the cold war was ending, Updike’s everyman was not alone in feeling at a loss. By the 1980s, the cold war had become more than a mere situation or circumstance. It was a state of mind.

Most Americans had come to take its existence for granted. Like the polar ice cap or baseball’s status as the national pastime, it had acquired an appearance of permanence. So its passing caught citizens unaware. Those charged with managing the cold war were, if anything, even more surprised. The enterprise to which they had devoted their professional lives had suddenly vanished. Here was a contingency that the sprawling US national security apparatus, itself a product of the anti-communist crusade, had failed to anticipate.

Freedom without constraints: how the US squandered its cold war victory – podcast Read more

On one level, of course, the surprise could not have been more gratifying. In the epic competition pitting west against east, the god-fearing against the godless and democracy against totalitarianism, “our side” had won. All-out nuclear war had been averted. The cause of freedom, which Americans felt certain they themselves embodied, had prevailed. Victory was decisive, sweeping and unequivocal.

In another sense, however, the passing of the cold war could not have been more disorienting. In 1987, Georgi Arbatov, a senior adviser to the Soviet leader Mikhail Gorbachev, had warned: “We are going to do a terrible thing to you – we are going to deprive you of an enemy.”

As the Soviet Union passed out of existence, Americans were left not just without that enemy, but without even a framework for understanding the world and their place in it. However imperfectly, the cold war had, for several decades, offered a semblance of order and coherence. The collapse of communism shattered that framework. Where there had been purposefulness and predictability, now there was neither.

Winning the cold war brought Americans face-to-face with a predicament comparable to that confronting the lucky person who wins the lottery: hidden within a windfall is the potential for monumental disaster. Putting that money to good use while avoiding the pitfalls inherent in suddenly acquired riches calls for prudence and self-awareness – not easily demonstrated when the big house, luxury car and holiday home you have always wanted are yours for the asking.

Similarly, the end of the cold war might have given Americans pause, especially since the issues at hand were of considerably greater significance than homes and cars. At least in theory, the moment might have invited reflection on some first-order questions, such as: what is the meaning of freedom? What does freedom allow? What obligations does it impose? Whom or what does it exclude?

Of course, Americans had been wrestling with such questions since well before 1776, the answers evolving over time. During the several decades of the cold war, however, the exigencies of the east-west rivalry had offered a reason to throttle down impulses to explore freedom’s furthermost boundaries. Except on the fringes of American politics, most citizens accepted the word from Washington that their way of life was under grave threat. In the pecking order of national priorities, addressing that threat – defending freedom rather than enlarging it – tended to take precedence over other considerations.

This is not to suggest that cold-war Americans were a compliant lot. They were not. From the 1950s, misleadingly enshrined as a decade of conformity, through the Ronald Reagan-dominated 80s, domestic crises and controversies were constants. Among the issues energising or enraging Americans were civil liberties, the nuclear arms race, mismanaged wars of dubious provenance, challenges to artistic tradition, leftwing and rightwing radicalism, crass materialism that coexisted with widespread poverty and a host of simmering issues connected to race, sex and gender. Yet through it all, a common outlook, centred on resistance to the “red threat”, endured. For most citizens most of the time, the cold war itself sufficed to explain “the point of being an American”.

The collapse of the Soviet empire between 1989 and 1991 robbed that outlook of its last vestiges of authority. Rarely, if ever, had the transition from one historical period to another occurred quite so abruptly, with such a precise set of demarcations, and with such profound implications. As if in an instant, the discipline that the cold war had imposed vanished. The absurdity of defining reality as an either/or choice – red or dead, slave or free, good v evil – now became blazingly apparent. The impact on American ambitions and expectations was akin to removing the speed limiter from an internal combustion engine. Suddenly the throttle opened up. The future appeared uniquely promising, offering Americans a seemingly endless array of choices, while confronting them with few evident constraints. Everything seemed possible.

Confident that an era of unprecedented US economic, military and cultural ascendancy now beckoned, members of an intoxicated elite threw caution to the winds. They devised – and promulgated – a new consensus consisting of four elements.

Facebook Twitter Pinterest Germans on top of the Berlin Wall in front of the Brandenburg Gate on 10 November 1989. Photograph: Fabrizio Bensch/Reuters

The first of these was globalisation or, more precisely, globalised neoliberalism. Stripped to its essence, globalisation was all about wealth creation: unconstrained corporate capitalism operating on a planetary scale in a world open to the movement of goods, capital, ideas and people would create wealth on a hitherto unimagined scale.

The second element was global leadership, a euphemism for hegemony or, more simply still, for empire. At its core, global leadership was all about order: unchallengeable military might would enable the US to manage and police a postcolonial yet implicitly imperial order favourable to American interests and values. Through the exercise of global leadership, the US would enforce globalisation. Order and abundance would go hand in hand.

The third element of the consensus was freedom, an ancient word now drastically revised. The new conception of freedom emphasised autonomy, with traditional moral prohibitions declared obsolete and the removal of constraints maximising choice. Order and abundance together would underwrite freedom, relieving Americans of existential concerns about safety and survival to which those less privileged were still obliged to attend.

The final element of the consensus was presidential supremacy, with the occupant of the Oval Office accorded quasi-monarchical prerogatives and status. Implicit in presidential supremacy was a radical revision of the political order. While still treated as sacred writ, the constitution no longer described the nation’s existing system of governance. Effectively gone, for example, was the concept of a federal government consisting of three equal branches. Ensuring the nation’s prosperity, keeping Americans safe from harm, and interpreting the meaning of freedom, the president became the centre around which all else orbited, the subject of great hopes, and the target of equally great scorn should he fail to fulfil the expectations that he brought into office.

All these elements together constituted a sort of operating system. The purpose of this operating system, unseen but widely taken for granted, was to cement the primacy of the US in perpetuity, while enshrining the American way of life as the ultimate destiny of humankind. According to the calendar, the end of the 20th century, frequently referred to as the American century, was then drawing near. Yet with the cold war concluding on such favourable terms, the stage appeared set for a prolonged American epoch.

This, however, was not to be.

The US wasted little time in squandering the advantages it had gained by winning the cold war. Events at home and abroad put the post-cold war consensus to the test, unmasking its contradictions and exposing its premises as delusional. Although globalisation did enable some to acquire great wealth, it left behind many more, while fostering egregious inequality. The assertion of global leadership provided American soldiers with plentiful opportunities to explore exotic and unfamiliar lands, but few would mistake the results for even an approximation of dominion, much less peace and harmony.

Instead, Americans came to accept war as habitual. And while the drive for individual empowerment removed constraints, it did little to promote the common good. An enlarged conception of freedom brought with it a whiff of nihilism. As for exalting the chief executive as a visionary leader, it yielded a succession of disappointments, before imploding in November 2016.

The post-cold war moment, dating from the early 90s and spanning the administrations of Bill Clinton, George W Bush and Barack Obama, turned out to be remarkably brief. By 2016, large numbers of ordinary Americans had concluded, not without reason, that the post-cold war consensus was irretrievably defective. Globalised neoliberalism, militarised hegemony, individual empowerment and presidents elevated to the status of royalty might be working for some, but not for them. They also discerned, again not without cause, that establishment elites subscribing to that consensus, including the leaders of both political parties, were deaf to their complaints and oblivious to their plight.

Get the Guardian’s award-winning long reads sent direct to you every Saturday morning

By turning their country over to Donald Trump, those Americans signalled their repudiation of that very consensus. That Trump himself did not offer anything remotely like a reasoned alternative made his elevation to the presidency all the more remarkable. He was a protest candidate elected by a protest vote. In that regard, the 2016 presidential election marked a historical turning point comparable in significance to the fall of the Berlin Wall a quarter of a century earlier.

As the cold war had evolved from the late 40s into the 80s, the rhetoric of freedom remained central to American political discourse. Among members of the intelligentsia, fads came and went, but none displaced freedom as the defining issue of the age. As the designated “leader of the free world”, each US president was in turn expected to talk the talk. From Truman through to Reagan, with differing levels of eloquence, none failed to do so.

The way it ended – with euphoric young Germans dancing on the wall – imparted to the entire cold war a retrospective moral clarity that it did not deserve. The cold war tainted everything it touched. As an episode in world history, it was a tragedy of towering proportions. So its passing ought to have called for reflection, remorse, repentance, even restitution. Yet the prevailing mood allowed for none of these, at least as far as most Americans were concerned. Instead, out of an era punctuated throughout by anxiety and uncertainty came a sense that a dazzling future lay just ahead.

In effect, the passing of the cold war relieved Americans of any further obligation to exhibit more than nominal cohesion. Except as a matter of personal preference, virtues such as self-discipline and self-denial, once deemed essential to enabling a nation to stand firm against existential threats, now became passé. The spirit of the post-cold war era prioritised self-actualisation and self-indulgence over self-sacrifice.

The demise of communism removed the last remaining constraints on the operation of global capitalism. By leaving the US militarily pre-eminent, the end of the cold war removed any remaining constraints on the use of American coercive power. Similarly, for many ordinary Americans, particularly those of a progressive bent, the passing of the cold war did away with any lingering constraints on matters related to “lifestyle”. No longer would they defer to the customary arbiters of propriety and “good taste” in determining what was permissible and what was not. For transcendent authority, progressives looked to the autonomous self.

‘At the heart of liberty is the right to define one’s own concept of existence, of meaning, of the universe, and of the mystery of human life.” So wrote supreme court justice Anthony Kennedy in a famous decision handed down shortly after the cold war ended.

Kennedy’s reformulation of liberty, however grandiose, was well suited to the mood that swept through elite quarters at the end of history. By comparison, the inalienable rights specified in the famous Declaration of Independence in 1776 now seemed cramped, stingy and inadequate. Freedom was in line for a makeover.

The emerging post-cold war conception of freedom was nothing if not expansive. It recognised few limits and imposed fewer obligations, with one notable exception: compliance was non-negotiable. As always, the American definition of liberty, however recently revised, was universally applicable, as valid in Bogotá and Dakar as it was in Boston and Denver.

What did this signify in practice? Allowing individuals maximum latitude to reach their own conclusions regarding the concepts of existence, meaning, the universe and the mystery of human life yielded what sort of society? The quarter-century that elapsed between the fall of the Berlin Wall and Donald Trump’s election provided a tentative answer to that question. Part of that answer came in the form of progress towards eliminating the remaining vestiges of racism, empowering women and reducing discrimination experienced by LGBTQ+ Americans.

Granted, progress does not imply decisive and irreversible success. Yet during the post-cold war period, American society became more tolerant, more open, more accepting and less judgmental. Attitudes toward people of colour, women and gays that in the 50s had been normative and remained widespread in the 60s and 70s had, by 2016, become unacceptable in polite society. Yet for more than a few Americans, Justice Kennedy’s notion of liberty as an opportunity to ponder life’s ultimate questions had little relevance. In practical terms, the exercise of freedom, undertaken in an environment in which consumption and celebrity had emerged as preeminent values, encouraged conformity rather than independence. At least notionally, Americans now enjoyed more freedom than ever before. Yet from every direction, but especially from Madison Avenue, Hollywood and Silicon Valley, came cues for how to make the most of the freedom now on offer. And however much you had, you always needed more.

Facebook Twitter Pinterest President Ronald Reagan (R) with Soviet leader Mikhail Gorbachev in Washington in December 1987. Photograph: Gary Hershorn/Reuters

So along with freedom came stress, anxiety and a sense of not quite measuring up, or a fear of falling behind as the demands of daily life seemed to multiply. For some, freedom meant alienation, anomie and despair. It did nothing to prevent, and in some instances arguably fostered, self-destructive or antisocial behaviour.

So in 2016, as another presidential election approached, Americans were able to claim the following distinctions:

• One in six were taking prescription psychiatric drugs such as antidepressants or anti-anxiety meds.

• More than 16 million adults and more than 3 million adolescents were suffering from significant depression.

• More than 1.9 million Americans were regularly using cocaine, with a half million hooked on heroin and 700,000 on methamphetamine.

• That year opioid overdoses killed 46,000, a new record.

• Binge drinking had reached epidemic proportions, with one in six US adults binge drinking several times a month and consuming seven drinks per binge; according to the Centers for Disease Control and Prevention, bingeing was especially common among younger and more affluent Americans.

• Nearly 45,000 were taking their own lives annually, the national suicide rate increased by 24% since 1999; within the previous decade the suicide rate of teenage girls had doubled and of boys had jumped by 40%.

• Smartphone addiction was joining more traditional compulsions, with the average person checking their smartphone 110 times a day, impelled by Fomo – a fear of missing out.

• Compulsive-buying syndrome, AKA shopping addiction, afflicted an estimated 6% of the population; a comparable number were compulsive hoarders.

• On a daily basis, 11 million Americans, mostly women, struggled with eating disorders such as anorexia and bulimia, while roughly 40% of adults and nearly 19% of children and adolescents were obese.

• Cosmetic surgeons were performing more than 17m procedures annually, with buttock augmentation and labiaplasty enjoying a particular spike in popularity.

• Forty-million Americans were regularly visiting online porn sites.

• The number of Americans infected with sexually transmitted diseases in 2016 surpassed 2 million, according to the CDC, “the highest number ever”.

• An estimated 24.7 million children were growing up in fatherless households, with such children substantially more likely to drop out of school, abuse drugs and alcohol and kill themselves; girls raised without a father present were four times more likely to get pregnant as teenagers.

• Although difficult to quantify with precision, 676,000 American children in 2016 were victims of abuse or neglect.

• Exercising their right to choose, American women were terminating around 650,000 unwanted pregnancies each year, despite the widespread availability of contraceptives.

• Exercising their right to bear arms, Americans had accumulated more than 40% of the planet’s small arms; the US arsenal in private hands was larger than that of the next 25 countries combined.

• Meanwhile, more than 33,000 Americans were being killed in firearms-related incidents annually.

• Year in and year out, the US had the world’s highest incarceration rate, no other developed nation coming anywhere close.

•Polling data showed that social trust – how Americans felt about government institutions and their fellow citizens – had sunk to an all-time low. Perhaps for that reason, when it came to voting, most Americans couldn’t be bothered; voter turnout in the US lagged behind that of most other developed countries.

• In an increasingly networked society, with two-thirds of Americans on Facebook, chronic loneliness afflicted a large portion of the population.

• In a phenomenon described as “deaths by despair”, the life expectancy of white working-class American males was dropping, a trend without historical precedent.

• The nation’s birthrate had fallen below the rate needed to sustain a stable overall population; America had ceased to reproduce itself.

• Not to be overlooked, in their pursuit of life, liberty, and happiness, Americans were polluting, wasting food and generating trash with abandon, leading the world in each category.

Arguably, Americans were enjoying more freedom than ever. Were they happier as a consequence? Polls suggested otherwise. In the 2007 “world happiness” standings, the US had ranked third among developed countries. By 2016, its position had plummeted to 13th.

By no means am I suggesting that a single such statistic holds the key to assessing life in the US. It does not. Nor do the various penchants and pathologies enumerated above. Yet taken together, they suggest a society in which discontent, dysfunction and sheer perversity were rampant.

As with globalised neoliberalism, some Americans not only coped with seemingly limitless freedom, but also luxuriated in the opportunities that it offered. For sophisticates inhabiting Brooklyn’s Park Slope, radical autonomy could well prove to be a boon; for those stuck in a ghetto on Chicago’s South Side, not so much. As for the accompanying underside, those in possession of sufficient resources could insulate themselves from its worst effects – just as the affluent were able to insulate themselves from the accumulating post-cold war military misadventures by simply allowing others to shoulder the burden.

What role did Trump play in shaping this US that worked nicely for some while leaving many others adrift and vulnerable? None at all. Globalisation, the pursuit of militarised hegemony, a conception of freedom conferring rights without duties, and a political system centred on a quasi-monarchical chief of state each turned out to have a substantial downside. Yet the defects of each made their appearance well before Trump’s entry into politics, even if elites, held in thrall by the post-cold war outlook, were slow to appreciate their significance. None of those defects can be laid at his feet.

If anything, Trump himself had displayed a considerable aptitude for turning such defects to his own advantage. In the US, post-cold war, he was prominent among those who enriched themselves, lived large and let others do the dirty work, while also shielding themselves from the difficulties that made life a trial for many of their fellow citizens. In an era of con artists, cowards and cynics, Trump became a modern equivalent of showman PT Barnum, parlaying the opportunities at hand into fortune, celebrity, lots of golf, plenty of sex and eventually the highest office in the land.

Yet for our purposes, the key point is this: Trump did not create the conditions in which the campaign of 2016 was to take place. Instead, to a far greater extent than any of his political rivals, he demonstrated a knack for translating those conditions into votes. Here, the moment met the man.

Trump’s critics saw him as an abomination. Perhaps he was. Yet he was also very much a man of his time. In the end, what won him the presidency was his capacity to push the buttons of millions of voters who believed themselves ill-served and left behind – abandoned, even – by establishment politicians of both parties.

Implicit in his promise to “make America great again” was an admission that greatness itself, which Americans had long since come to believe was theirs by right, had been lost, with no one taking responsibility and no one, apart from Trump himself, venturing to explain how it had even happened. The critical word that imparted to his campaign slogan its formidable persuasive power was “again”. As Tom Engelhardt has written, it represented an acknowledgment that self-congratulatory terms such as “great”, “super”, “exceptional” or “indispensable” no longer reflected the actually existing American condition. Millions of ordinary citizens recognised this as self-evidently true. Arrangements, agreements and advantages that Americans had once prized had been squandered or thrown away. And yet no politician other than Trump dared to utter that truth aloud.

As a strategic thinker, Trump had no particular talent. Yet as a strategic sensor, he was uniquely gifted, possessing an intuitive genius for reading the temper of his supporters and stoking their grievances. Yet by no means did Trump create those grievances – they had festered during the quarter-century after the cold war ended. He merely recognised their existence and, in doing so, made himself the champion of the aggrieved and the one person they came to believe who might respond to their plight.

The post-cold war recipe for renewing the American century has been tried and found wanting. A patently amoral economic system has produced neither justice nor equality, and will not. Grotesquely expensive and incoherent national security policies have produced neither peace nor a compliant imperium, and will not. A madcap conception of freedom unmoored from any overarching moral framework has fostered neither virtue nor nobility nor contentment, and won’t anytime soon. Sold by its masterminds as a formula for creating a prosperous and powerful nation in which all citizens might find opportunities to flourish, it has yielded no such thing. This, at least, describes the conclusion reached by disenchanted Americans in numbers sufficient to elect as president someone vowing to run the post-cold war consensus through a shredder.

Donald Trump’s detractors charge him with dividing the country when, in fact, it was pervasive division that vaulted him to the centre of American politics in the first place. The divide is deepest and least reconcilable between those Americans for whom the trajectory of events since the cold war pointed upward, and those who found in those same events evidence of decline and decay, and who sensed they had been had. At the most fundamental level, the inhabitants of one camp believe that talent, skills and connections will enable them to determine their own destiny; they are masters of their own fate. In the other camp are those who see themselves as victims. As Obama put it while campaigning for the presidency in 2008, “they cling to guns or religion or antipathy toward people who aren’t like them, or anti-immigrant sentiment or anti-trade sentiment as a way to explain their frustrations”.

Trump did not create this cleavage. He merely turned it to his personal advantage. So, regardless of the date or terms of Trump’s departure from office, the schism that allowed him to become president is likely to persist after he is gone. It’s that schism, rather than the antics of the tycoon/reality TV star/demagogue who exploited it, that merits far more attention than it has received.

This is an edited extract from The Age of Illusions: How America Squandered Its Cold War Victory by Andrew Bacevich, published by Metropolitan Books/Holt

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.