Harold Davies didn't cry this time. Four years ago when I accompanied him to the polls his eyes welled up as he described how it felt to vote for an African American candidate. This time he was in and out within 10 minutes and then off to his brother's for his tea. You can only elect the first black president once. To use the euphoria of 2008 against the more toned-down celebrations of Tuesday night as a stick to beat Barack Obama misunderstands the significance of his trajectory.

Electing a black candidate on his promise, amid a massive economic crisis, is one thing. To re-elect him on his record, even as that crisis endures, is quite another. In more ways than one his victory on Tuesday night was more impressive than in 2008 precisely because it was not more symbolic.

It's difficult to think of a more vulnerable president facing re-election and pulling it off so decisively. Having redrawn the electoral map and reshaped the electorate in 2008 he managed to give a plausible account of his efforts over the past four years even when they had fallen short. His fallibility as a candidate is now accepted; his timidity as a leader now beyond question.

On a flight to Denver last week an Obama supporter sitting next to me explained how his view of the president had evolved: "I thought he was a prophet. Now I realise he's just a king." Sooner or later he will have to get used to the fact that his president is just a human being.

There was a lot riding on Obama's victory even as there was precious little expected of it. It is easy to forget the state the country was in both domestically and internationally four years ago and easier to overlook how Obama's first term made a concrete difference to people's lives. Last year Chelsea Shinneman of Roanoke, Virginia, had a baby, Harrison, who was born with a congenital heart defect that would have condemned him to a lifetime of high health insurance premiums were it not for Obamacare. In northern Colorado 36 families at risk of homelessness were helped by stimulus money. In Sarasota, Florida, Betty-Jean Haines took a picture of her grandson in his army uniform out of her wallet and said: "Thanks to Obama, he's coming home."

But over the last month I've met very few, even among his most ardent supporters, who believed much would change if he won. For even as the nation kept the president in place it also kept the Republicans in charge of the House of Representatives and the Democrats in charge of the Senate. In other words for all the billions spent, speeches given, doors knocked and phones called, it woke up on Wednesday morning with exactly the same leadership it had on Tuesday night. And taken as a whole it's not a leadership people are particularly impressed by. The high turnout signified interest in the presidential race and improved organisation on the ground. But it didn't indicate a growing confidence in the political culture.

"We are an American family and we rise or fall together as one nation and as one people," Obama said in his victory speech. But to much of the country it looks like a dysfunctional family over which they have no control. Obama's electoral college victory was emphatic but on the popular vote the country remains evenly divided and fundamentally split. It was telling in the runup to election day how even the pollsters and the polls themselves became politicised even though they were deadly accurate. The two sides can't agree on the most basic facts. It's difficult to reach across the aisle when the width and position of the aisle are in dispute.

When I asked Michelle Venus in Fort Collins, Colorado, a few weeks ago if she thought things would improve for her if Obama won, she shrugged. Venus is a diehard Democrat who went from having a $75,000-a-year job, her own home, savings and a pension to visiting a food bank in just a couple of years thanks primarily to the recession and two bereavements.

"Obama's tried," she said. "But all I see is a Congress that is hellbent on getting him out of office. The system is developed to work carefully, slowly and thoughtfully, but I don't see the thoughtful happening very much. If nothing else happens Obama will at least keep the status quo."

Mark Weaver, once the chairman of Colorado's Loveland chamber of commerce, who also ended up visiting the food bank for a short while, held out little hope that either candidate would be up to the job. "Somebody's got to fix the economy, but I don't know that either of them has the guts to do it," he said. "I'm looking to vote for someone I like and trust; I've never been more distrustful of the whole thing. I wish we could vote for none of the above. I want a do-over." These are among the many people for whom change is necessary but who see no real sign of it through politics.

There are a few reasons to believe that this might change. The first is that the Republican party has reached a point where it will either have to change or die. This election effectively exposed it as a mono-racial party in an increasingly multi-racial state. At every rally you can see it. Regardless of the ethnic composition of the area in which they are held, the composition of rallies never changes. At the Republican convention one person threw peanuts and insults at a black camerawoman. The Grand Old Party is becoming the White People's Party. And that is not only unbecoming, it is untenable.

Every month 50,000 new Latinos become eligible to vote. What Tuesday night showed was that the new coalition Obama cohered in 2008 that mobilised the young, the brown and the black in unprecedented numbers was not just a one-off. Soon, North Carolina, Arizona and ultimately Texas will be tough to hold if Republicans refuse to challenge the xenophobia of their base.

"This election is a critical juncture for the Republican party," says Christine Sierra, who heads UNM's Southwest Hispanic Research Institute. "Latinos would be a more diverse voting bloc if there were reasons to vote Republican. But without those reasons it's possible they could become as cohesive a voting bloc as African Americans."

For the best part of half a century the Republicans have invoked Richard Nixon's Southern strategy to prise suburban and southern whites away from the Democrats. "You have to face the fact that the whole problem is really the blacks," Nixon told his chief-of-staff. "The key is to devise a system that recognises that while not appearing to."

This made electoral (if not moral) sense as recently as the 90s. But since 1980 the white share of the vote has fallen in every election bar one and over the last four election cycles fewer and fewer people of colour will have voted Republican. That leave Republicans fishing for a larger number of voters in a smaller pool of white people. Take Nevada, once a reliably Republican state. In 2008 Blacks and Latinos made up 25% of the voting population; this time they made up 27%. And 91% of African Americans and 70% of Latinos voted for Obama.

"This is the last time anyone will try to do this," one Republican strategist told the National Journal during the convention when Romney was polling in the high 20s. Back then the Republican consultant Ana Navarro told the Los Angeles Times: "Where his numbers are right now, we should be pressing the panic button." On Tuesday night he got 21% of the Latino vote.

The other reason is that, like in 2010, the Republicans forfeited the opportunity to take the US Senate by selecting hardline conservatives who said and did things that offended many and, in particular, deeply alienated women. This was particularly evident in Indiana where Republicans ousted a long-term conservative, who would have been a shoo-in, for a Tea Party candidate. Obama lost the state by 10 points. The Democrat won the Senate seat by six points. Women made the difference, with 52% voting for the Democratic Senate candidate and for Mitt Romney. After 2008 there was little reflection within the Republican party as to the source of their defeat, or the question of how the country had got into war, recession, debt and despair under eight years of their leadership. Instead they doubled down on an extremist agenda that has alienated more than it has galvanised. Given the state of the economy and Obama's inability to deliver to his base this was their race to lose.

The easiest thing for them to do would be to blame Romney for their defeat. But for all his faults – and there were many – it was the party that was the problem. Romney only started to be viable as a candidate when he sounded like a moderate and ran on his record in Massachusetts, where he was governor. His problem was that in order to be viable within the Republican party he had to sound like an extremist and run away from his record. The result was not just an uninspiring candidate – the nation can live with that – but an unconvincing one just one YouTube mash-up away from being exposed as opportunistic and inauthentic.

This was Obama's gain, but it was America's loss. On a range of issues that Americans care deeply about he should have been challenged. The past four years could have been better. All too often he approached the financial industry, the health insurance industry, the stimulus battle and his Republican opponents in Congress like he did Romney in the first debate. Rather than take the fight to them he receded and allowed them to define him and his agenda. The result has been a "recovery" in which the banks are healthy and the nation still sick, most of the jobs that have been created are low-paid and the biggest winners of his healthcare reform are in the insurance companies.

So long as the Republicans were so extreme and Obama was vulnerable – which was for far more of the campaign than many realise – this was not a conversation most Democrats were prepared to have. Every attempt to raise a substantive policy point would be shot down by electoral realities. What you would like to happen was always eclipsed by the question: "Who would you rather win?"

The people who delivered him a second term on Tuesday night – the young, black and brown – are those who were least likely to have benefited from his first. His greatest cheer for the night was when he reprised a version of the speech that made him famous when he introduced John Kerry at the Democratic convention in 2004.

"I believe we can keep the promise of our founders," he said. "The idea that if you're willing to work hard, it doesn't matter who you are or where you come from or what you look like or where you love. It doesn't matter whether you're black or white or Hispanic or Asian or Native American or young or old or rich or poor, able, disabled, gay or straight, you can make it here in America if you're willing to try."

That is what people voted for. The trouble is that while Obama has the eloquence to articulate those aspirations he has, for the most part, not had the courage to fight for them or the strategic ability to deliver them.

"If you're going to be president then I guess you obviously want to be in the history books," said Susan Aylward in Akron, Ohio, who voted for Obama. "So what does he want to be in the history books for? I don't quite know the answer to that yet." Perhaps in the next four years we'll find out.

This article originally appeared on guardian.co.uk