To be at the Democratic National Convention in Denver in August 2008 was a little like attending a religious revivalist meeting. Nearly eight years of George W Bush and his toxic friends – the Cheneys and Rumsfelds who dragged America’s international reputation below even where it had been in the depths of the Vietnam War – had left Americans humiliated and baying for something better. In Barack Obama they seemed to have found it, with the added bonus, over 40 years after the civil rights movement supposedly achieved its aims, of proving that one no longer needed to be a white man to lead the most powerful nation on Earth. It was a new dawn: but the day has not been quite so sunlit as was hoped.

In America last week I encountered one person after another who, to different degrees and using different means of expression, had had enough. It appears to be an exceptionally unhappy country: polarised, introspective, angry, disappointed and, above all, fearful. We have seen many representations and manifestations of American angst and rage in recent years: the horrific and depressingly frequent shootings in schools, racially motivated attacks, powder kegs in cities which ignite in confrontations between the police and minorities. Yet all of these, however pernicious, are largely transient for those not directly involved. The sense of grievance dominating this year’s presidential election campaign has cooked more slowly, is more deep-seated, and will require more than simply a new president – of whatever stamp – to appease it.

As far as Americans are concerned, some of the disappointment with the Obama administration must be considered self-inflicted: they have elected a Republican-dominated Congress that has thwarted much of what he sought to do. Yet Obama, too, must bear some blame. His inexperience – he was a one-term senator before securing his party’s nomination eight years ago – has hindered him, in his handling of Congress and of issues alike. His attempt to reform health care met difficulties of implementation unconnected with the visceral opposition of Republicans and, indeed, of some Democrats. His elevated rhetoric has too often preceded a lower reality.

He is perceived to have disengaged from a world in which many Americans of all political persuasions feel their country’s influence could once more be beneficial, if asserted sensibly. But, above all, the Obama years have done little to improve the struggle of tens of millions of Americans who live at a crushing level of poverty and amid a dereliction that shocks many western Europeans who see it.

Perhaps it was inevitable that a country defined by its ability to create opportunity and provide success should, at some point, pause for breath, take stock and ­allow doubt to seep in; and it is right that the foreign policy disasters of the Bush years should cause it to reflect deeply before engaging with the world. But all this has sapped national self-confidence and verve, creating negativity that seeks, and finds, expression in the election campaign and which all the candidates, in their various ways, are trying to reflect or exploit. The mood threatens fundamentally to change America’s politics.

It is hard to believe that in any other ­contest since the civil war, such an outsider as Donald Trump, who has never served in the military or held political office, would have neared the point of securing the nomination of a major political party – and, if he does, that he would have more of a prospect than many would like to imagine of securing the presidency, and with it the so-called leadership of the free world.

That Trump seems to have said things in private which contradict what he has said in public may be reassuring as far as his ­perceived extremism goes, but it says little for his honesty. He refuses to publish his tax returns, which raises questions about his financial probity, already under the spotlight because of the fiasco of the erstwhile Trump University, where students have parted with sums in excess of $40,000 in return for worthless qualifications. There is also speculation that he refuses to publish his returns because he has exaggerated his charitable donations.

More worrying is Trump’s foreign policy, and his access to the nuclear button. The effect on the Middle East of implementing his belligerent ideas for dealing with Isis can only be a matter of anxious conjecture; as must be his potential relations with his fellow narcissist and megalomaniac Vladimir Putin.

However, America’s current state of mind and sense of betrayal by mainstream politicians also account for the enthusiasm shown for the attempt by Bernie Sanders, the self-declared socialist from Vermont, to wrest the Democratic Party’s nomination from Hillary Clinton. Any attempt to equate Sanders with Jeremy Corbyn flatters Corbyn: Sanders is more eloquent, more thoughtful, and infinitely more credible. He looks unlikely to succeed, but his support among younger voters especially demonstrates the growing sense of a need among Americans, as they look ahead, to break a cycle of disappointment and failure. It is interesting that one important part of the senator’s platform – the anathematising of free trade – coincides with an obsession of Trump’s: a political analyst told me he would not be at all surprised if some of Sanders’s supporters voted Trump in the event of Sanders not winning the Democratic nomination.

“After all,” he said, without a trace of humour, “they’re both Democrats.”

***

Donald Trump’s success seems to rest on two things: his populist appeal, which includes a nod to the xenophobia of America’s less educated masses and plays to their fears more generally, and the fact that although he has dabbled in politics for many years (he donated to Hillary Clinton’s campaign in 2008, like any good Democrat), he is not a politician. Bernie Sanders shares that with him: although a long-term congressman, he is hardly of the political class as Americans understand it.

As in Britain, it is common and easy in America today to view the political class with contempt. But that has long been the case – covering Bob Dole’s failed campaign to beat Bill Clinton 20 years ago I saw a T-shirt that proclaimed “Death Is No Excuse: Nixon in ’96”. Now, however, the sense of contempt has reached crisis point.

Recent research shows that there has been no improvement in the real wages of working-class and lower-middle-class Americans since the late 1970s; they are below what they were in the late 1960s and early 1970s. America is a country of astonishing wealth, but this is concentrated in the hands of a tiny group among its population of nearly 325 million. Park Avenue and Rodeo Drive are very much the exceptions, not the rule. In New York there seem to be more homeless people on the streets than I remember seeing since the late 1980s. There are large industrial wastelands around many of America’s cities: much has been written about how huge areas of Detroit have been put back to grass, but dereliction afflicts substantial parts of many cities. The country has over $19trn of public debt, and it is hard to see where all the cash is going, other than, as one disaffected Democrat told me, “on the military”. The US spends roughly $598bn a year on defence (3.3 per cent of its GDP in 2015), four times as much as China (1.2 per cent of its GDP last year), seven times as much as Saudi Arabia (12.9 per cent) and ten and a half times as much as the UK (2 per cent).

Other measures of living standards are telling. The 2012 census found that 48 million Americans, or 15.4 per cent of the population, were uninsured for health care. Levels of achievement for school-leavers are lower than in most developed countries, and a lower proportion of Americans graduates from higher education than in comparable nations. Perhaps it is an indication of the national unhappiness, as well as low income, that medical surveys report that 17 per cent of children are obese and 32 per cent overweight. Black and Hispanic students in the US rank the lowest in assessments for maths, science, reading and problem-solving, but American students overall rank poorly internationally. The US is one of only three OECD countries (the other two are Turkey and Israel) that spends more on schools in rich areas than in poor ones.

Housing is the most obvious sign of poverty and decay. In 2013, the government’s department of housing and urban development reported that the average annual income for a resident of a public housing project was $13,370. The proportion of people renting rather than owning is increasing, driving up rents. The level of home ownership is below where it was when Bill Clinton launched a national drive to encourage ownership in the late 1990s, and thanks not least to the effects of the sub-prime crisis related to that policy. Evictions, especially of black and Hispanic families, are on the rise and in many towns the evictions industry – bailiffs, removal men, lawyers – is a major employer. In New York, New Jersey and California, 22 per cent of households spend more than 50 per cent of their pre-tax income on housing. With many people all over the country unable to afford either to rent or to buy, the number of 25-year-olds living with their parents is also rising.

Even travelling around a city as affluent as New York, you see the signs of decay everywhere. A colleague told me of the rats she sees in the street on her way to and from work. The subway stations are dirty and basic. The roads are potholed and infrastructure generally is creaking: the lack, for instance, of a modern public transport link between the city and its main international airport, JFK, is absurd. Many buildings, even in Manhattan, show want of repair. Utilities run inefficiently: a friend’s cooking gas supply in the East Village failed last September and has only just been fixed. The signs of extreme affluence and modernity are ubiquitous, too, which only makes the portents of a decaying civilisation more disturbing.

***

The widespread poverty is demoralising enough for Americans to witness, but the prevailing mood is driven by something less tangible, and that is fear. As Trump’s remarkably successful campaign seems to have proved, many Americans are fearful of attack by Islamic extremists. His bizarre promise to ban Muslims from the United States may appear entirely ludicrous – and an alleged secret tape recording of him with editorial staff of the New York Times supposedly has him admitting it – yet it has scored a direct hit with millions of his fellow countrymen. So has his proposal to build a wall to stop illegal immigrants entering from Mexico, because so many people in that land of immigrants fear the effect of an unregulated wave of them. And, flowing from that, they fear further extension of those industrial wastelands, with their ruined factories and warehouses, because of the Chinese undercutting them, which is why they like Trump’s threat to provoke a suicidal trade war with China.

There is no simple reason for why things have gone so badly wrong. It is certainly not for want of good intentions. The practicalities of governing so vast a country to deliver standards of living expected in a developed Western nation, even under a federal system spread across 50 states, have perhaps become too much, and create huge scope for corruption and waste. I recall being told by a distinguished Washington journalist during the Clinton years that America was ungovernable and would have to split into four or five separate countries. There is still no sign of that, yet.

Automation threw millions of Americans out of manufacturing and into service industries, if they went into any work at all. The education system fails many of them. With 70 per cent still not holding a passport, their country remains one that understands less about the world than it ought, and learns few lessons from the rest of it. There are pockets of sweetness and light, yet much of America appears to be in a dark and recriminatory cul-de-sac. It sounded ridiculous for Hillary Clinton, on Super Tuesday, to call again for more “love and kindness” in America, but she had a point. It seems more the inclination of many Americans to look first for their enemies rather than their friends.

An American friend talked to me of his country having some sort of collective nervous breakdown. It is tempting to cite that to explain the rise of Donald Trump, but it would be wrong. Many Americans who would like him to win know exactly what they are doing, as do others who, in a different way, wish to break the system by choosing Bernie Sanders. There is a widespread view that the usual solutions will not suffice to put America back on track, and that without some degree of desperate measures the decline of America will become even steeper. The greatest fear of all seems to be fear of the future.

Simon Heffer is a columnist for the Daily and Sunday Telegraphs