The town of Pella, Iowa, looks an almost too perfect vision of smalltown America. Surrounded by a chessboard of prosperous farmland and with a bustling town square, lined with shops bearing the surnames of its first Dutch settlers, Pella feels like a throwback to a different age.

But beneath its attractive exterior last week one could find some ugly sentiments on election day. "Obama is a Muslim," said Shirley Schutte, 75. Was she sure about that? "I am. I am not sure he even should have been there [in the White House]. He has been a disaster."

Such a fervent belief is not typical of most Republican voters, whether in Pella or anywhere else in America. But it is not hard to find. One poll in Mississippi even found some 52% of likely Republican voters suspected President Barack Obama was a follower of Islam. Neither has the party leadership done too much to discourage equally outlandish ideas, such as Obama being born in Kenya. From business mogul Donald Trump to top elected officials, Republicans have carefully crafted a message of Obama as a radical "other" hoping to transform America in some dangerous way.

Yet far from exiling Obama outside the US mainstream, many experts, now including leading conservative figures, believe the Republican party itself is being pushed into the political wilderness. The Republicans increasingly look like the party of angry, older white people. People like Schutte. And that does not work in America any more.

US election voting Photograph: NTC Economics

As Republicans sifted through the wreckage of the Mitt Romney campaign, they saw collapsing popularity among fast-emerging ethnic groups, such as Hispanics, and key social demographics, such as young people. In an economy struggling with 7.9% unemployment, where more than half of voters believed the country was heading in the wrong direction and against an unpopular incumbent, the once fiercely effective Republican party machine only managed to craft a devastating defeat.

Some say the reason is a simple failure to change in an America that is becoming less white and more socially liberal. "They look a lot more like a political party of the 1950s than a party of the 21st century," said Professor David Cohen, a political scientist at the University of Akron in Ohio. "They are at risk of being irrelevant."

Some in the party know it. Even though the corpse of the defeated Romney campaign is still warm, a bitter fight has started to break out over its meaning in Republican ranks. On one side are the modernists, who understand that the party cannot afford to be seen as a backwards-looking ghetto for white voters. On the other are the nativists, angry at a crippled and ineffective immigration system, who believe that only a true message of pure conservatism will save the day. It is a battle for the soul of the Republican party and the first shots are being fired. "I think it is going to be a war. I really do," said Larry Haas, a political commentator and former aide in the Clinton White House.

Last week the Romney campaign in the key swing state of Iowa held a "victory" party in the capital, Des Moines. Right in the American heartland, in the very state that gave birth to Obama's presidential ambitions in 2008, the great and good of the local Republican party gathered in a downtown hotel ballroom to celebrate their side's expected win.

But shortly after the local TV station announced Obama had won Iowa – in the end by a hefty six percentage points – Fox News said that the White House also would remain in Democratic hands. The mood of the almost entirely white gathering of several hundred rapidly deflated. Some headed to the exits. One woman muttered angrily to her companion: "It is the dumbing down of America."

This is the side of the Republican party that has dominated its internal politics for four years. It is a party that almost seems to exist in its own vacuum of rightwing thought. Infused with Tea Party radicals, it has backed hardline immigration laws in states such as Arizona that many Hispanics see as racist. It boasted two Senate candidates who made tone-deaf comments about rape that cost them otherwise easy victories. It is still male-dominated, yet finds time to take hardline ideological stances on female contraception and abortion. This is the party that appears implacably hostile to gay Americans even as last week four more states held ballots on gay marriage and all voted in favour. "Does social conservatism continue to be a albatross around the neck of the party?" said Professor Gerard Alexander of the conservative American Enterprise Institute.

But it is not just social issues. On economics the Republican party plays host to a powerful and vocal wing of libertarians who wish to slash and burn government spending. They cling to a conservative world view that has forced previously extreme stances – such as abolishing the federal Department of Education and returning the dollar to the gold standard – into the heart of Republican thought. Not even the vast amount of cash that Republican big money operators poured into the 2012 race was able to have a major impact. Of the top 10 Senate candidates that political guru Karl Rove's American Crossroads group spent the most on, just one resulted in a Democratic defeat. Casino billionaire Sheldon Adelson backed eight candidates – including Romney – with around $60m over the whole election cycle. None of them won.

To many observers, the Republicans are turning into a party that cannot win office. It has been dominated by the punditocracy of Fox News and the enormous influence of rightwing media stars such as Glenn Beck and Rush Limbaugh. It believes it does not need to change, but must maintain ideological purity and run a true conservative candidate. In Romney it sees the failure of a moderate who did not really believe the conservative values he had to espouse to win his party's nomination.They point out Obama's victory was built on a superior ground game, which turned out its base. They can even say Obama only beat Romney by 50% to 48% – a sliver that only grows large in the undemocratic electoral college.

Washington Post columnist Charles Krauthammer has emerged as one of the leading lights of this message. "The answer to Romney's failure is not retreat, not apeing the Democrats' patchwork pandering," he thundered. "No whimpering. No whining. No reinvention when none is needed. Do conservatism, but do it better."

Limbaugh was more blunt. "I went to bed last night thinking we're outnumbered. I went to bed last night thinking we'd lost the country," he told listeners as Romney went down. But perhaps Fox News host Bill O'Reilly – for many fans the very incarnation of the average white man – was the most blatant: "The white establishment is now the minority … it's not a traditional America any more."

But many are lining up on the other side of the trenches. Indeed, even Krauthammer acknowledges the party has a serious problem with Hispanic voters, who now make up the fastest-growing part of the electorate and went for Obama by some 70%. These are people such as Texas senator Ted Cruz and Florida senator Marco Rubio, who has already announced his intention to visit Iowa this month, effectively firing the first shot of the 2016 campaign. They also include former Florida governor Jeb Bush, whose last name is still a political handicap but whose Hispanic wife, half-Hispanic children and fluent Spanish are a major asset to dragging Republicans out of their white corner.

As such figures rise, and perhaps bring with them a greater sensitivity over issues such as immigration, they will strike a blow for the reformers and the party's makeup will come to better represent the wider American public. Yet it might not be that simple. In an economy still struggling with high joblessness and the threat of renewed recession still looming, convincing some of the party's stressed base might not be easy. "The backroom people in the party look at the numbers and know they have a problem. But it is another thing to convince the base," said Professor Shaun Bowler, a political scientist at the University of California at Riverside.

Neither is it as easy as just shifting the ethnic tone of the party's public image. Many Republican activists say that Hispanics – who often display a strong social conservatism around Roman Catholicism – should find a natural home in the party. However, many also bring with them a profoundly different sense of the role of government. The hostility many in the Republican party express towards government programmes can be just as off-putting to many Hispanic voters as their opposition to abortion and gay marriage might be attractive.

It is not likely to be an easy process. Some believe Romney came close enough to victory to allow an even fight in the coming Republican civil war and thus ensure a protracted and painful debate that will stretch on for years. What the party really needed, some think, was to have nominated a died-in-the-wool ultra-conservative in 2012 such as Rick Santorum or Newt Gingrich who could have led the party to an overwhelming defeat, forcing the reformist wing to triumph. But the selection of Romney denied them that piece of creative destruction, even though the party has now lost the popular vote tally in five of the last six presidential elections. "They are still maybe at the early stages of denial," Bowler said.

Democrats are largely celebrating the prospect of this fight. The glee among the liberal left has been unrestrained, ranging from serious political pundits, such as MSNBC host Rachel Maddow and film-maker Michael Moore, to the viral popularity of an internet site showing photographs of sad Republicans on election night called "White People Mourning Romney".

The Democrats, in fact, are licking their lips at the prospect of the next four years. Obama's brilliant strategists have created a highly effective coalition of minorities, younger women voters and urban educated people. They eked out an election win in the most trying of economic circumstances by getting those people to the polls. But some people think the Democrats also have a problem. Obama lost the white vote in America by some 20 points, and perhaps that should not be ignored. "It is not good to lose the white vote by that margin," said Haas. "This election was visionless on both sides, it was just about stitching together enough votes to get to the top."

The Republicans may be about to have a civil war over their future but the Democrats also have their issues when it comes to the full spectrum of America's broad and diverse electorate. When any political system fights over identity politics rather than actual ideas, no one really wins.