In the New York Times earlier this year, Paul Krugman wrote of an economic effect called "The Great Gatsby curve," a graph that measures fiscal inequality against social mobility and shows that America's marked economic inequality means it has correlatively low social mobility. In one sense this hardly seems newsworthy, but it is telling that even economists think that F Scott Fitzgerald's masterpiece offers the most resonant (and economical) shorthand for the problems of social mobility, economic inequality and class antagonism that we face today. Nietzsche – whose Genealogy of Morals Fitzgerald greatly admired – called the transformation of class resentment into a moral system "ressentiment"; in America, it is increasingly called the failure of the American dream, a failure now mapped by the "Gatsby curve".

Fitzgerald had much to say about the failure of this dream, and the fraudulences that sustain it – but his insights are not all contained within the economical pages of his greatest novel. Indeed, when Fitzgerald published The Great Gatsby in April 1925, the phrase "American dream" as we know it did not exist. Many now assume the phrase stretches back to the nation's founding, but "the American dream" was never used to describe a shared national value system until a popular 1917 novel called Susan Lenox: Her Fall and Rise, which remarked that "the fashion and home magazines … have prepared thousands of Americans … for the possible rise of fortune that is the universal American dream and hope." The OED lists this as the first recorded instance of the American dream, although it's not yet the catchphrase as we know it. That meaning is clearly emerging – but only as "possible" rise of fortune; a dream, not a promise. And as of 1917, at least some Americans were evidently beginning to recognise that consumerism and mass marketing were teaching them what to want, and that rises of fortune would be measured by the acquisition of status symbols. The phrase next appeared in print in a 1923 Vanity Fair article by Walter Lippmann, "Education and the White-Collar Class" (which Fitzgerald probably read); it warned that widening access to education was creating untenable economic pressure, as young people graduated with degrees only to find that insufficient white-collar jobs awaited. Instead of limiting access to education in order to keep such jobs the exclusive domain of the upper classes (a practice America had recently begun to justify by means of a controversial new idea called "intelligence tests"), Lippmann argued that Americans must decide that skilled labour was a proper vocation for educated people. There simply weren't enough white-collar jobs to go around, but "if education could be regarded not as a step ladder to a few special vocations, but as the key to the treasure house of life, we should not even have to consider the fatal proposal that higher education be confined to a small and selected class," a decision that would mark the "failure of the American dream" of universal education.

These two incipient instances of the phrase are both, in their different ways, uncannily prophetic; but as a catchphrase, the American dream did not explode into popular culture until the 1931 publication of a book called The Epic of America by James Truslow Adams, which spoke of "the American dream of a better, richer and happier life for all our citizens of every rank, which is the greatest contribution we have made to the thought and welfare of the world. That dream or hope has been present from the start. Ever since we became an independent nation, each generation has seen an uprising of ordinary Americans to save that dream from the forces that appear to be overwhelming it."

In the early years of the great depression Adams's book sparked a great national debate about the promise of America as a place that fosters "the genuine worth of each man or woman", whose efforts should be restricted by "no barriers beyond their own natures". Two years later, a New York Times article noted: "Get-rich-quick and gambling was the bane of our life before the smash"; they were also what caused the "smash" itself in 1929. By 1933, Adams was writing in the New York Times of the way the American dream had been hijacked: "Throughout our history, the pure gold of this vision has been heavily alloyed with the dross of materialistic aims. Not only did the wage scales and our standard of living seem to promise riches to the poor immigrant, but the extent and natural wealth of the continent awaiting exploitation offered to Americans of the older stocks such opportunities for rapid fortunes that the making of money and the enjoying of what money could buy too often became our ideal of a full and satisfying life. The struggle of each against all for the dazzling prizes destroyed in some measure both our private ideals and our sense of social obligation." As the Depression deepened, books such as Who Owns America? A New Declaration of Independence were arguing that "monopoly capitalism is morally ugly as well as economically unsound," that in America "the large majority should be able – in accordance with the tenets of the 'American dream' … to count on living in an atmosphere of equality, in a world which puts relatively few barriers between man and man." Part of the problem, however, was that the dream itself was being destroyed by "the friends of big business, who dishonour the dream by saying that it has been realised" already.

The phrase the American dream was first invented, in other words, to describe a failure, not a promise: or rather, a broken promise, a dream that was continually faltering beneath the rampant monopoly capitalism that set each struggling against all; and it is no coincidence that it was first popularised during the early years of the great depression. The impending failure had been clear to Fitzgerald by the time he finished Gatsby – and the fact that in 1925 most Americans were still recklessly chasing the dream had a great deal to do with the initial commercial and critical failure of The Great Gatsby, which would not be hailed as a masterpiece until the 50s, once hindsight had revealed its prophetic truth.

On 19 October 1929, just five days before the first stock market crash and 10 days before Black Tuesday, Scott Fitzgerald published a now-forgotten story called "The Swimmers," about an American working for the ironically named Promissory Trust Bank, and his realisation that American ideals have been corrupted by money. This corruption is emblematised by sexual infidelity: as in Gatsby, Fitzgerald again used adultery to suggest a larger world of broken promises and betrayals of faith. There's a remarkable moment early in "The Swimmers" – which Fitzgerald called "the hardest story I ever wrote, too big for its space" – when an unfaithful wife, who is French, complains about the American women she sees on the Riviera:

"How would you place them?" she exclaimed. "Great ladies, bourgeoises, adventuresses - they are all the same. Look! …"

Suddenly she pointed to an American girl going into the water:

"That young lady may be a stenographer and yet be compelled to warp herself, dressing and acting as if she had all the money in the world."

"Perhaps she will have, some day."

"That's the story they are told; it happens to one, not to the ninety-nine. That's why all their faces over thirty are discontented and unhappy."

The American dream comes true for just 1%: for the other 99%, only discontent and bitterness await, ressentiment on a mass scale. More than 15 years later, the Marxist critics Theodor Adorno and Max Horkheimer used a similar image of the typist who believed she would be a movie star to reveal the American dream as a rigged lottery that no one wins but everyone plays. Today, almost 100 years after "The Swimmers" appeared, the Occupy movement has clenched its fist around the same angry realisation that we are all the 99%, not the 1%. More remarkable than the fact that Fitzgerald beat Adorno and Horkheimer and the Occupy movement to the punch, however, is that he saw all this before Wall Street came smashing down.

The villain of "The Swimmers" is a rich, vulgar banker who preaches an updated version of the gilded age's "gospel of wealth": "Money is power … Money made this country, built its great and glorious cities, created its industries, covered it with an iron network of railroads." The banker is wrong, the story makes clear, but his vision of America is winning. Feeling increasingly alienated, the protagonist, Marston, finds himself musing on the meanings of America, and especially its eagerness to forget history: "Americans, he liked to say, should be born with fins, and perhaps they were – perhaps money was a form of fin. In England property begot a strong place sense, but Americans, restless and with shallow roots, needed fins and wings. There was even a recurrent idea in America about an education that would leave out history and the past, that should be a sort of equipment for aerial adventure, weighed down by none of the stowaways of inheritance or tradition." The buoyancy of modern America depended on its being unanchored by history or tradition, and this is the America we have inherited. Historical amnesia is certainly liberating – so liberating that America is once again diving into free fall, unmoored by any critical or intellectual insight into its own myths, or even into the histories of the debates that we think define our moment.

Marston eventually decides that there is no place for him in the crass society symbolised by his rival, but he will not relinquish his faith in the ideals that America can represent. As Marston sails for Europe, watching America recede into his past, Fitzgerald offers a closing meditation nearly as incantatory as the famous conclusion of Gatsby: "Watching the fading city, the fading shore, from the deck of the Majestic, he had a sense of overwhelming gratitude and of gladness that America was there, that under the ugly débris of industry the rich land still pushed up, incorrigibly lavish and fertile, and that in the heart of the leaderless people the old generosities and devotions fought on, breaking out sometimes in fanaticism and excess, but indomitable and undefeated. There was a lost generation in the saddle at the moment, but it seemed to him that the men coming on, the men of the war, were better; and all his old feeling that America was a bizarre accident, a sort of historical sport, had gone forever. The best of America was the best of the world … France was a land, England was a people, but America, having about it still that quality of the idea, was harder to utter – it was the graves at Shiloh and the tired, drawn, nervous faces of its great men, and the country boys dying in the Argonne for a phrase that was empty before their bodies withered. It was a willingness of the heart."

Wall Street crashed 10 days later.

❦

Two years after The Great Gatsby appeared, a reporter was sent to interview the famous author. Meeting "the voice and embodiment of the jazz age, its product and its beneficiary, a popular novelist, a movie scenarist, a dweller in the gilded palaces", the reporter found instead, to his distinct hilarity, that Fitzgerald was "forecasting doom, death and damnation to his generation". "He sounded", said the reporter, like "an intellectual Sampson" predicting that the Plaza Hotel's marble columns would crumble. Fitzgerald's absurd prophecy was that America would face a great "national testing" in the very near future:

"The idea that we're the greatest people in the world because we have the most money in the world is ridiculous. Wait until this wave of prosperity is over! Wait ten or fifteen years! Wait until the next war on the Pacific, or against some European combination! … The next fifteen years will show how much resistance there is in the American race."

"There has never been an American tragedy," Fitzgerald ended. "There have only been great failures."

It was 1927. The reporter was vastly amused.

• Sarah Churchwell's Careless People, on the origins of The Great Gatsby, will be published in 2013.