The final chapter of America's Promise, a high-school textbook on American history, ends with a rallying cry to national mythology. "The history of the United States is one of challenges faced, problems resolved, and crises overcome," it states. "Throughout their history Americans have remained an optimistic people, carrying this optimism into the new century. The full promise of America has yet to be realised. This is the real promise of America; the ability to dream of a better world to come."

Such are the assumptions beamed from the torch of Lady Liberty, coursing through the veins of the nation's political culture and imbibed with mothers' milk. Their nation, many will tell you, is not just a land mass but an ideal – a shining city on the hill beckoning a bright new tomorrow and a dazzling dawn for all those who want it badly enough. Such devout optimism, even (and at times particularly) in the midst of adversity makes America, in equal parts, both exciting and delusional. According to Gallup, since 1977 people have consistently believed their financial situation will improve next year even when previous years have consistently been worse.

But when President Barack Obama was planning his run for a second term his pollsters noticed a profound shift in the national mood. The optimism was largely gone – and with it both the excitement and the delusion. The time-honoured rhetorical appeals to a life of relentless progress, upward mobility and personal reinvention didn't work the way they used to.

"The language around the American dream wasn't carrying the same resonance," Joel Benenson, one of Obama's key pollsters, told the Washington Post. "Some of the symbols of achieving the American dream were becoming burdens – owning that house with the big mortgage was expensive, owning two cars and more debts; having your kid go to college. The cost and burden of taking out those loans was making a lot of Americans ambivalent. They weren't sure a college education was worth it."

This wasn't just about the recession – though of course that didn't help – but a far more protracted, profound and painful descent in expectations and aspirations that has been taking place for several decades. For underpinning that faith in a better tomorrow was an understanding that inequality in wealth would be tolerated so long as it was coupled with a guarantee of equality of opportunity. In recent years they have seen both heading in the wrong direction – the gap between rich and poor has grown even as possibilities for economic and social advancement have stalled.

Between 2007 and 2010 the median American family lost a generation of wealth, putting them on a par with where they were in 1992. Last week the census revealed that median household income is roughly the same as it was in 1988 and that the poverty rate had actually increased since 1973. Meanwhile, median male earnings in 2010 were on a par with 1964. This is not for want of effort. American workers continue to make gains in productivity and American companies continue to reap the benefits. Last year corporate profits, as a share of the economy, were the highest since the second world war. The trouble is, none of the benefits went back to them.

And while wages have stagnated and wealth has dissipated, costs have shot up. A family's health insurance contributions have increased 90% over the past decade. Over the past five years tuition costs have leapt 27% at state universities and 13% at private institutions above inflation. A nation that has long prided itself on being forward-thinking is reconciling itself to going backwards.

A Heartland Monitor poll last week showed that two-thirds of adults believe their children will enjoy less financial security than they do and face more challenges than opportunities. Last year the same pollsters found a slim majority defining getting ahead as simply "not falling behind". This lack of confidence in the ability to get ahead is twinned with almost total despair that the political class has neither the means nor the will to reverse the trend.

A Pew poll this month showed that they believed the government policies since the recession had helped the rich, banks and corporations and done nothing for the poor, the middle class or small businesses. They are right to be sceptical.

With the budget battle and raising the debt limit looming, the narrative will once again be that America's deadlocked political culture is crippling its economy. It's true that it certainly doesn't help. But it has done far more damage when its components have worked together.

For the root causes of this decline are not difficult to fathom. "One of the biggest drivers is deindustrialisation," explains James Meadway, senior economist at the New Economics Foundation. "Manufacturing used to provide good paying secure jobs for skilled workers in the west but much of that work has gone to emerging economies."

Both parties supported the financial deregulation and international trade liberalisation that laid the foundations for this despondency. Each blames the other for its consequences. Neither is capable of doing anything about it because the very monied interests that made this situation possible also make the politicians.

Little of this is unique to the US. The main countries within the European Union, including Britain, are suffering from wage stagnation, creating a squeezed middle class. The retraction has not been taking place as long as it has in the US, but has been on the same trajectory for quite some time. But there are two key differences. The Germans, British and French were not raised with a guiding myth that is being contradicted at the end of every month when families routinely find they can't meet their expenses. Second, most European nations do have a welfare state (at least for now) that contends with the fallout. Those who look to America as a trendsetter that maps our future should be careful what they wish for.

The self-proclaimed leader of the free world is turning into a low-wage economy with a class system more rigid than most and a middle class that wavers between poverty and precariousness. More than half the people using the food bank in Larimer County, Colorado, that I visited last year were working. More than one in four families in New York's homeless shelters includes at least one working adult. In the absence of a living wage and an ethical pay structure, the work ethic, on which the American dream is founded, doesn't work.