California has a special place in the American psyche. It is the Golden State: a playground of the rich and famous with perfect weather. It symbolises a lifestyle of sunshine, swimming pools and the Hollywood dream factory.

But the state that was once held up as the epitome of the boundless opportunities of America has collapsed. From its politics to its economy to its environment and way of life, California is like a patient on life support. At the start of summer the state government was so deeply in debt that it began to issue IOUs instead of wages. Its unemployment rate has soared to more than 12%, the highest figure in 70 years. Desperate to pay off a crippling budget deficit, California is slashing spending in education and healthcare, laying off vast numbers of workers and forcing others to take unpaid leave. In a state made up of sprawling suburbs the collapse of the housing bubble has impoverished millions and kicked tens of thousands of families out of their homes. Its political system is locked in paralysis and the two-term rule of former movie star Arnold Schwarzenegger is seen as a disaster – his approval ratings having sunk to levels that would make George W Bush blush. The crisis is so deep that Professor Kevin Starr, who has written an acclaimed history of the state, recently declared: "California is on the verge of becoming the first failed state in America."

Outside the Forum in Inglewood, near downtown Los Angeles, California has already failed. The scene is reminiscent of the fallout from Hurricane Katrina, as crowds of impoverished citizens stand or lie aimlessly on the hot tarmac of the centre's car park. It is 10am, and most have already been here for hours. They have come for free healthcare: a travelling medical and dental clinic has set up shop in the Forum (which usually hosts rock concerts) and thousands of the poor, the uninsured and the down-on-their-luck have driven for miles to be here.

The queue began forming at 1am. By 4am, the 1,500 spaces were already full and people were being turned away. On the floor of the Forum, root-canal surgeries are taking place. People are ferried in on cushions, hauled out of decrepit cars. Sitting propped up against a lamp post, waiting for her number to be called, is Debbie Tuua, 33. It is her birthday, but she has taken a day off work to bring her elderly parents to the Forum, and they have driven through the night to get here. They wait in a car as the heat of the day begins to rise. "It is awful for them, but what choice do we have?" Tuua says. "I have no other way to get care to them."

Yet California is currently cutting healthcare, slashing the "Healthy Families" programme that helped an estimated one million of its poorest children. Los Angeles now has a poverty rate of 20%. Other cities across the state, such as Fresno and Modesto, have jobless rates that rival Detroit's. In order to pass its state budget, California's government has had to agree to a deal that cuts billions of dollars from education and sacks 60,000 state employees. Some teachers have launched a hunger strike in protest. California's education system has become so poor so quickly that it is now effectively failing its future workforce. The percentage of 19-year-olds at college in the state dropped from 43% to 30% between 1996 and 2004, one of the highest falls ever recorded for any developed world economy. California's schools are ranked 47th out of 50 in the nation. Its government-issued bonds have been ranked just above "junk".

Some of the state's leading intellectuals believe this collapse is a disaster that will harm Californians for years to come. "It will take a while for this self-destructive behaviour to do its worst damage," says Robert Hass, a professor at Berkeley and a former US poet laureate, whose work has often been suffused with the imagery of the Californian way of life.

Now, incredibly, California, which has been a natural target for immigration throughout its history, is losing people. Between 2004 and 2008, half a million residents upped sticks and headed elsewhere. By 2010, California could lose a congressman because its population will have fallen so much – an astonishing prospect for a state that is currently the biggest single political entity in America. Neighbouring Nevada has launched a mocking campaign to entice businesses away, portraying Californian politicians as monkeys, and with a tag-line jingle that runs: "Kiss your assets goodbye!" You know you have a problem when Nevada – famed for nothing more than Las Vegas, casinos and desert – is laughing at you.

This matters, too. Much has been made globally of the problems of Ireland and Iceland. Yet California dwarfs both. It is the eighth largest economy in the world, with a population of 37 million. If it was an independent country it would be in the G8. And if it were a company, it would likely be declared bankrupt. That prospect might surprise many, but it does not come as news to Tuua, as she glances nervously into the warming sky, hoping her parents will not have to wait in the car through the heat of the day just to see a doctor. "It is so depressing. They both worked hard all their lives in this state and this is where they have ended up. It should not have to be this way," she says.

It is impossible not to be impressed by the physical presence of Arnold Schwarzenegger when he walks into a room. He may appear slightly smaller than you imagine, but he's just as powerful. This is, after all, the man who, before he was California's governor, was the Terminator and Conan the Barbarian.

But even Schwarzenegger is humbled by the scale of the crisis. At a press conference in Sacramento to announce the final passing of a state budget, which would include billions of dollars of cuts, the governor speaks in uncharacteristically pensive terms. "It is clear that we do not know yet what the future holds. We are still in troubled waters," he says quietly. He looks subdued, despite his sharp grey suit and bright pink tie.

Later, during a grilling by reporters, Schwarzenegger is asked an unusual question. As a gaggle of journalists begins to shout, one man's voice quickly silences the others. "Do you ever feel like you're watching the end of the California dream?" asks the reporter. It is clearly a personal matter for Schwarzenegger. After all, his life story has embodied it. He arrived virtually penniless from Austria, barely speaking English. He ended up a movie star, rich beyond his dreams, and finally governor, hanging Conan's prop sword in his office. Schwarzenegger answers thoughtfully and at length. He hails his own experience and ends with a passionate rallying call in his still thickly accented voice.

"There is people that sometimes suggest that the American dream, or the Californian dream, is evaporating. I think it's absolutely wrong. I think the Californian dream is as strong as ever," he says, mangling the grammar but not the sentiment.

Looking back, it is easy to see where Schwarzenegger's optimism sprung from. California has always been a special place, with its own idea of what could be achieved in life. There is no such thing as a British dream. Even within America, there is no Kansas dream or New Jersey dream. But for California the concept is natural. It has always been a place apart. It is of the American West, the destination point in a nation whose history has been marked by restless pioneers. It is the home of Hollywood, the nation's very own fantasy land. Getting on a bus or a train or a plane and heading out for California has been a regular trope in hundreds of books, movies, plays, and in the popular imagination. It has been writ large in the national psyche as free from the racial divisions of the American South and the traditions and reserve of New England. It was America's own America.

Michael Pollan, author of The Omnivore's Dilemma and now an adopted Californian, remembers arriving here from his native New England. "In New England you would have to know people for 10 years before they let you in their home," he says. "Here, when I took my son to his first play date, the mother invited me to a hot tub."

Michael Levine is a Hollywood mover and shaker, shaping PR for a stable of A-list clients that once included Michael Jackson. Levine arrived in California 32 years ago. "The concept of the Californian dream was a certain quality of life," he says. "It was experimentalism and creativity. California was a utopia."

Levine arrived at the end of the state's golden age, at a time when the dream seemed to have been transformed into reality. The 1950s and 60s had been boom-time in the American economy; jobs had been plentiful and development rapid. Unburdened by environmental concerns, Californian developers built vast suburbs beneath perpetually blue skies. Entire cities sprang from the desert, and orchards were paved over into playgrounds and shopping malls.

"They came here, they educated their kids, they had a pool and a house. That was the opportunity for a pretty broad section of society," says Joel Kotkin, an urbanist at Chapman University, in Orange County. This was what attracted immigrants in their millions, flocking to industries – especially defence and aviation – that seemed to promise jobs for life. But the newcomers were mistaken. Levine, among millions of others, does not think California is a utopia now. "California is going to take decades to fix," he says.

So where did it all go wrong?

Few places embody the collapse of California as graphically as the city of Riverside. Dubbed "The Inland Empire", it is an area in the southern part of the state where the desert has been conquered by mile upon mile of housing developments, strip malls and four-lane freeways. The tidal wave of foreclosures and repossessions that burst the state's vastly inflated property bubble first washed ashore here. "We've been hit hard by foreclosures. You can see it everywhere," says political scientist Shaun Bowler, who has lived in California for 20 years after moving here from his native England. The impact of the crisis ranges from boarded-up homes to abandoned swimming pools that have become a breeding ground for mosquitoes. Bowler's sister, visiting from England, was recently taken to hospital suffering from an infected insect bite from such a pool. "You could say she was a victim of the foreclosure crisis, too," he jokes.

But it is no laughing matter. One in four American mortgages that are "under water", meaning they are worth more than the home itself, are in California. In the Central Valley town of Merced, house prices have crashed by 70%. Two Democrat politicians have asked for their districts to be declared disaster zones, because of the poor economic conditions caused by foreclosures. In one city near Riverside, a squatter's camp of newly homeless labourers sleeping in their vehicles has grown up in a supermarket car park – the local government has provided toilets and a mobile shower. In the Los Angeles suburb of Pacoima, one in nine homeowners are now in default on their mortgage, and the local priest, the Rev John Lasseigne, has garnered national headlines – swapping saving souls to saving houses, by negotiating directly with banks on behalf of his parishioners.

For some campaigners and advocates against suburban sprawl and car culture, it has been a bitter triumph. "Let the gloating begin!" says James Kunstler, author of The Long Emergency, a warning about the high cost of the suburban lifestyle. Others see the end of the housing boom as a man-made disaster akin to a mass hysteria, but with no redemption in sight. "If California was an experiment then it was an experiment of mass irresponsibility – and that has failed," says Michael Levine.

Nowhere is the economic cost of California's crisis writ larger than in the Central Valley town of Mendota, smack in the heart of a dusty landscape of flat, endless fields of fruit and vegetables. The town, which boldly terms itself "the cantaloup capital of the world", now has an unemployment rate of 38%. That is expected to rise above 50% as the harvest ends and labourers are laid off. City officials hold food giveaways every two weeks. More than 40% of the town's people live below the poverty level. Shops have shut, restaurants have closed, drugs and alcohol abuse have become a problem.

Standing behind the counter of his DVD and grocery store, former Mendota mayor Joseph Riofrio tells me it breaks his heart to watch the town sink into the mire. His father had built the store in the 1950s and constructed a solid middle-class life around it, to raise his family. Now Riofrio has stopped selling booze in a one-man bid to curb the social problems breaking out all around him.

"It is so bad, but it has now got to the point where we are getting used to it being like this," he says. Riofrio knows his father's achievements could not be replicated today. The state that once promised opportunities for working men and their families now promises only desperation. "He could not do what he did again. That chance does not exist now," Riofrio says.

Outside, in a shop that Riofrio's grandfather built, groups of unemployed men play pool for 25 cents a game. Near every one of the town's liquor stores others lie slumped on the pavements, drinking their sorrows away. Mendota is fighting for survival against heavy odds. The town of 7,000 souls has seen 2,000 people leave in the past two years. But amid the crisis there are a few sparks of hope for the future. California has long been an incubator of fresh ideas, many of which spread across the country. If America emerges from its crisis a greener, more economically and politically responsible nation, it is likely that renewal will have begun here. The clues to California's salvation – and perhaps even the country as a whole – are starting to emerge.

Take Anthony "Van" Jones, a man now in the vanguard of the movement to build a future green economy, creating millions of jobs, solving environmental problems and reducing climate change at a stroke. It is a beguiling vision and one that Jones conceived in the northern Californian city of Oakland. He began political life as an anti-poverty campaigner, but gradually combined that with environmentalism, believing that greening the economy could also revitalise it and lift up the poor. He founded Green for All as an advocacy group and published a best-selling book, The Green Collar Economy. Then Obama came to power and Jones got the call from the White House. In just a few years, his ideas had spread from the streets of Oakland to White House policy papers. Jones was later ousted from his role, but his ideas remain. Green jobs are at the forefront of Obama's ideas on both the economy and the environment.

Jones believes California will once more change itself, and then change the nation. "California remains a beacon of hope… This is a new time for a new direction to grow a new society and a new economy," Jones has said.

It is already happening. California may have sprawling development and awful smog, but it leads the way in environmental issues. Arnold Schwarzenegger was seen as a leading light, taking the state far ahead of the federal government on eco-issues. The number of solar panels in the state has risen from 500 a decade ago to more than 50,000 now. California generates twice as much energy from solar power as all the other US states combined. Its own government is starting to turn on the reckless sprawl that has marked the state's development.

California's attorney-general, Jerry Brown, recently sued one county government for not paying enough attention to global warming when it came to urban planning. Even those, like Kotkin, who are sceptical about the end of suburbia, think California will develop a new model for modern living: comfortable, yes, but more modest and eco-friendly. Kotkin, who is writing an eagerly anticipated book about what America will look like in 2050, thinks much of it will still resemble the bedrock of the Californian dream: sturdy, wholesome suburbs for all – just done more responsibly. "We will still live in suburbs. You work with the society you have got. The question is how we make them more sustainable," he says.

Even the way America eats is being changed in California. Every freeway may be lined with fast-food outlets, but California is also the state of Alice Waters, the guru of the slow-food movement, who inspired Michelle Obama to plant a vegetable garden in the White House. She thinks the state is changing its values. "The crisis is bringing us back to our senses. We had adopted a fast and easy way of living, but we are moving away from that now," she says.

There is hope in politics, too. There is a growing movement to call for a constitutional convention that could redraw the way the state is governed. It could change how the state passes budgets and make the political system more open, recreating the lost middle ground. Recently, the powerful mayor of Los Angeles, Antonio Villaraigosa, signed on to the idea. Gerrymandering, too, is set to take a hit. Next year Schwarzenegger will take steps to redraw some districts to make them more competitive, breaking the stranglehold of party politics. He wants district boundaries to be drawn up by impartial judges, not politicians. In previous times that would have been the equivalent of a turkey voting for Christmas. But now the bold move is seen for what it is: a necessary step to change things. And there is no denying that innovation is something that California does well.

Even in the most deprived corners of the state there is a sense that things can still turn around. California has always been able to reinvent itself, and some of its most hardcore critics still like the idea of it having a "dream".

"I believe in California. It pains me at the moment to see it where it is, but I still believe in it," said Michael Levine.

Perhaps more surprisingly, a fellow believer is to be found in Mendota in the shape of Joseph Riofrio. His shop operates as a sort of informal meeting place for the town. People drop in to chat, to get advice, or to buy a cold soft drink to relieve the unrelenting heat outside. The people are poor, many of them out of work, often hiring a bunch of DVDs as a cheap way of passing the time. But Riofrio sees them as a community, one that he grew up in. He is proud of his town and determined to stick it out. "This is a good place to live," he says. "I want to be here when it turns around." He is talking of the stricken town outside. But he could be describing the whole state.★

• This article was amended on Monday 5 October 2009. We said Kenneth Starr had written a history of California. We meant Kevin Starr. The sub-heading of this article was also changed because state staff are not being paid in IOUs.